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Foreword 



Nothing is more exciting than when 
a student “gets it” — hones a new 
skill, makes an important connection, 
achieves deepened understanding, de- 
velops a true love of the subject. It’s the 
reward for everything we do. But find- 
ing out, with confidence and clarity, ex- 
actly what our students have learned is 
a tremendous challenge, and that’s 
where assessment comes in. 

The excitement of our adventure is 
that we don’t have all the answers yet! 
Higher education today faces many 
rising expectations: for student learn- 
ing, institutional effectiveness, account- 
ability, integrating classroom and out- 
of-class experiences, using instructional 
technology effectively, and delivering 
programs of outstanding quality and 
public credibility in a time of con- 
strained resources. Rising Expectations 
for Assessment: Can We Deliver?, the 
theme of AAHE’s 2000 Assessment 
Conference, captures the spirit of our 
quest for the best ways to meet rising 
expectations for higher education by as- 
sessing fairly and honestly and using 
our findings to improve what we do. 

The major speakers at the 2000 
Conference have laid the groundwork 
for our quest by giving us a strong un- 
derstanding of what we already know 
and the challenges that lie before us. 
None does this more effectively than 



by Linda Suskie 

Jorge Klor de Alva, now president and 
chief executive officer of the parent 
company of the University of Phoenix, a 
large and successful for-profit institu- 
tion. In a plenary session conversation 
with Gail Mellow, he explains how the 
University of Phoenix defines and as- 
sesses its learning outcomes and uses 
that information to develop its pro- 
grams. While some of those in tradi- 
tional higher education settings have 
concerns about the University of Phoe- 
nix’s approach, this new model will re- 
main a viable force, so it’s important, at 
the very least, to understand it, and 
perhaps to consider adapting some as- 
pects of its approach to assessment. 

Comedian Don Novello once did a 
bit about the “Five Minute University.” 
His premise was that we don’t remem- 
ber more than about five minutes’ 
worth of knowledge from our college 
education, so why not go to college for 
just five minutes? We want far more 
than that, of course; we want to pro- 
mote the deep, insightful learning that 
endures, and that’s why Noel Entwis- 
tle’s plenary remarks are vital to us all. 
He builds on the findings of years of re- 
search to give us valuable suggestions 
for ways to create curricula, pedagogies, 
and assessments that promote deep 
learning. 

It’s becoming increasingly clear 
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that treating students equitably does 
not mean treating them all the same, 
because students have varying back- 
grounds and temperaments that affect 
how they learn. James Anderson, key- 
note speaker for the conference’s “Ris- 
ing Expectations for Teaching and 
Learning’’ theme track, discusses cul- 
tural differences in learning styles and 
suggests ways to adapt our teaching 
and assessment strategies to meet the 
needs of diverse learners. 

One of the latest buzzwords for 
demonstrating institutional effective- 
ness is “performance indicators.” Victor 
Borden, keynote speaker for the “Rising 
Expectations for Institutional Effec- 
tiveness” track, draws on a book he 
coedited on performance indicators to 
share the characteristics of effective 
performance indicators and how to 
choose the best performance indicators 
for your particular situation. 

One of the exciting pedagogical de- 
velopments of the last few years has 
been the growth of the learning com- 
munities movement. Jean MacGregor, 
Vincent Tinto, and Jerri Holland Lind- 
blad, keynote speakers for the “Rising 
Expectations for Program Quality” 
track, share lessons learned from as- 
sessments of learning communities. 
From those lessons, they reveal the se- 
crets to assessing any innovative pro- 
gram successfully. 

The accreditation and assessment 
movements have had a collaborative 



relationship for many years. Barbara 
Wright, keynote speaker for the “Rising 
Expectations for Excellence in Assess- 
ment Practices” track, reviews that his- 
tory and offers suggestions on how we 
can use both assessment and accredita- 
tion to promote cultural change. 

As the late-night commercials say, 
“. . . but that’s not all!” This volume also 
includes highlights of Assessment as 
Evidence of Learning: Serving Student 
and Society, AAHE’s 1999 Assessment 
Conference. 

Tom Angelo, Peter Ewell, and 
Cecilia Lopez discuss three major forces 
having impacts on the assessment 
movement — accountability demands, 
growing diversification in postsecon- 
daiy education, and the new teach- 
ing/leaming paradigm — and they re- 
mind us of the lessons we’ve already 
learned. John Biggs echoes Noel 
Entwistle’s ideas on how our assess- 
ments influence how students study 
and learn and how we can promote 
“deep” learning. And Sharon Robinson 
analyzes several definitions of a “fair” 
test and offers suggestions on ways we 
can help all students demonstrate their 
understanding effectively. 

As we continue our quest for the 
best possible assessment models and 
tools, these talented, knowledgeable in- 
dividuals have laid a strong, thought- 
provoking foundation on which we can 
build. 



B 

Director of AAHE’s Assessment Forum at the time of its 2000 Assessment Conference, 
Linda Suskie is director of planning, assessment, and analysis at Millersville Uni- 
versity of Pennsylvania. r 
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A Conversation With 
Jorge Klor de Alva 

interviewed by Gail Mellow 



I n an article in Educause, Barry 
Munitz, former chancellor of the 
California State University system, 
talked about the way the family farm 
was changed by the introduction of the 
intercontinental railroad. Because of the 
change in delivery system, every form 
and function of the family farm 
changed. 

It’s a very apt analogy for what the 
University of Phoenix may be doing for 
higher education. The University 
blasted onto the radar screen of higher 
education about four or five years ago 
(even though it’s been around much 
longer) with a national curriculum in a 
for-profit setting, multiple sites, and a 
predominantly part-time faculty of prac- 
titioners. This very large and successful 
institution’s model has challenged many 
of our long-standing ideas about the 
form and function of higher education. 

Let’s begin, Jorge, with an overview 
of your vision for the University of 
Phoenix as an innovative campus in de- 
livering higher education. 

Before I say a few things about the 
University and its innovations, I want 
to qualify my statements with the fol- 



lowing: It’s important to understand 
that our mission is to focus on students 
who are at least 23 years old and who 
are employed. We deal exclusively with 
employed adults — working pro- 
fessionals — a sector of the student 
spectrum that is geared for the kind of 
education we provide. I don’t want to 
suggest that everything we do can be 
replicated in any setting. 

Given that proviso, what do we do 
and how do we do it? Since its begin- 
ning, the University has focused on do- 
ing research on what adults (students 
23 years old and older) need to be able 
to get into higher education and com- 
plete their studies. Through much trial 
and error, we developed a particular 
formula, which can be summarized as 
follows. 

First, our students want to be able 
to go to school while working full time. 
So our courses are taught year-round in 
a highly concentrated format, each 
course being either five or six weeks 
long. We can make our courses very 
concentrated because our students take 
only one course at a time. 

Second, discipline is critical if 
adults are to complete their education. 
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We know we must give our students an 
excuse to put milli ons of other far more 
important things aside so that they can 
get on with the business of their educa- 
tion. They need the excuse to be able to 
buckle down, and we had to assist them 
in developing it. So we set up a system 
of mandatory attendance. If you miss 
more than one class of a course at the 
University of Phoenix, no matter what 
the excuse, you are administratively 
dropped. If you’re an online student and 
you do not log on five out of seven days 
for meaningful work, no matter what 
the excuse, you’re administratively 
dropped. Raising the bar to failure with 
discipline is the “secret sauce” in our 
recipe, and that can be applied to many 
other settings. 

Third, our more than 7,100 faculty 
members, of whom some 240 are full- 
time, are overwhelmingly practitioners. 
They must be trained how to teach and, 
since they don’t have tenure, we’ve had 
to replace all the systems that are the 
positive sides of having tenure. We have 
replaced those systems through com- 
prehensive, continuous, and rigorous 
processes of assessment, beg innin g with 
the identification of potential faculty, 
during their training, and all through 
their career. They are not only con- 
tinually assessed but also given the 
tools to overcome whatever weaknesses 
they may have. 

The underlying assumption of the 
University of Phoenix model is that if 
you can’t measure something, you can- 
not manage it. In order to grow into a 
national university, we have had to 
know how to continually improve both 
academic and administrative process- 
es. We therefore believe that every- 
thing must be assessed on a regular ba- 
sis. In short, we measure — as we say 



— everything that moves, regularly and 
thoroughly. 

Let’s return to your target audience and 
your delivery mechanism for meeting 
their needs. Tell us more about how you 
did the assessment that allowed you to 
determine how to package the education 
you present to students. How did you 
learn how to package what I’d call the 
University of Phoenix “brand,” a distinc- 
tive way of offering education ? 

Before we do anything, we undertake 
studies. We tend to do relatively exten- 
sive, and frequently very expensive, 
studies before we enter a market, pro- 
duce a new program, or make signifi- 
cant changes within the institution. We 
continually conduct focus groups 
throughout the country to understand 
the issues we should be addressing. We 
repeatedly survey not only our students 
but also their employers. Before any- 
thing gets out the door, a lot of invest- 
ment has gone into figuring out what 
the real and perceived needs are — 
what we are really trying to address 
and the impact our moves will have. 

Let me give you an example. Be- 
fore we enter a market, we have a list of 
some 400 issues that must be re- 
searched. That includes everything that 
we can understand, not only about the 
demographic and economic setting but 
also about the technology infra- 
structure, the number and types of de- 
grees that people have, and the pro- 
grams already available in the region. 
We undertake a long, detailed analysis 
of all this information before we make a 
judgment about how we should respond 
to a particular market or curricular 
opportunity. 

Because we continually assess and 
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survey our students, all of whom are 
employed, and our faculty, most of 
whom are employed elsewhere, we’re 
continually gaining information about 
what employers need and therefore 
what employees need. That forms a big 
part of the beginning of any analysis of 
what programs we should develop. 

This is a very different model from 
traditional higher education. (I taught 
for twenty-seven years in traditional in- 
stitutions, so I am familiar with their 
secret sauces!) At traditional institu- 
tions, new programs often begin with 
very creative faculty members who de- 
velop them on their own and then take 
great pride in getting them through an 
understandably hesitant bureaucracy. 

At the University of Phoenix, new 
programs usually begin with faculty 
members who are “out there” and who 
tell us, “In our company we’re facing 
these kinds of problems. Is this true in 
companies all over the country?” Well, 
we go out and try to find out. And if it’s 
true that these problems are happening 
all over and they’re not being ad- 
dressed, the idea is put into the hopper, 
so to speak, to begin the process of 
analyzin g whether we should move in 
that direction or not. All of our pro- 
grams have been developed this way. 

We just developed over the last 
year a master’s and an undergraduate 
degree in e-business, and the whole 
process was done hand-in-hand with 
the employers who are absolutely cen- 
tral to the success of such a program. 
Our programs didn’t just come out of 
thin air. 

So few of us have the resources that you 
have as a for-profit. When you say you 
do very extensive and expensive assess- 
ments beforehand, how do you do them ? 



Do you hire research firms ? Do you have 
in-house assessment folks who do this? 

This kind of research usually begins 
within our own institutional research 
office. We have somewhere between 
twenty-five and thirty people working 
on institutional research issues, per- 
haps the largest institutional research 
group of any university. Their job is to 
do environmental scans as well as in- 
ternal assessments. 

In many cases, our staff isn’t large 
enough to do certain kinds of analyses, 
so we outsource those. When we come 
to Charlotte, for example, we will know 
a lot about Charlotte beforehand and 
we will do a fairly extensive market 
study. We will also work with our cam- 
puses in the nearby region and help fill 
them in on things we’ve discovered 
about Charlotte. We’re not always ca- 
pable of doing that in geographic areas 
where we don’t already have our people 
nearby. 

When we outsource studies, we 
crosscheck them. We put them through 
our own wringer to see if the findings 
really make sense, based on compara- 
tive analyses with other similarly situ- 
ated areas. 

Focus groups are another expense. 
We use focus groups for practically 
everything we do. 

Remember that this all stems from 
our philosophy: If you’re not measuring 
it or assessing it in some fashion, how 
would you know what policy to create 
around it? How would you know what 
decision to make? If anyone at the Uni- 
versity says, “Oh no, we can’t do that 
for this reason,” the first response of 
everyone else is, “Show me the data,” 
because we have data on just about 
everything of relevance to us. 
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How does the University of Phoenix as- 
sess student learning? 

While we focus on trying to do every- 
thing possible for students to have ac- 
cess to higher education, we do have a 
series of minimum standards that stu- 
dents must be able to meet in order to 
enter our undergraduate or graduate 
programs. 

Once a student meets our entrance 
requirements and enrolls, the student 
will begin his or her academic career at 
the University of Phoenix with a three- 
hour cognitive and affective assessment 
exam. We begin by assessing the stu- 
dents’ cognitive knowledge of the field 
they plan to study. Because we focus on 
practical skills — that’s why students 
come to us — we also assess them on 
critical thinking, communications skills, 
and those affective traits and qualities 
that are assumed to be key to success in 
their career. As a consequence, accord- 
ing to some researchers at Educational 
Testing Service (ETS), we have the 
largest database on adult education in 
the United States. 

Every single student is then as- 
sessed on the same things when he or 
she graduates: cognitive knowledge, af- 
fective traits, critical-t hinking skills , 
and communications s kills . In this way, 
we make sure that the money they 
spend going to the University of Phoe- 
nix and the time that they took away 
from their spouses, dogs, employers, 
and children was well spent, with a 
significant return on their (and for 
many, their employers’) investment. We 
can bring that information directly to 
the students and their employers. 
(Keep in mind that we work as closely 
with employers as with our students. 
That’s how we create effective systems 



and relevant curricula.) 

We also want to know how our stu- 
dents are doing compared with their 
peers in other systems. We therefore 
use the ETS’s Major Field Tests and 
Graduate Management Admissions 
Test (GMAT). 

While our aim is to have all our 
students take comprehensive exams at 
the beginning and end of their academic 
career, not all fields have national ex- 
ams at this time. Many of our students 
are in our MBA programs, for example, 
and there’s no national exam for an 
MBA today. Even in those programs for 
which national exams do exist, we ordi- 
narily can’t compare apples with ap- 
ples, meaning our employed adult stu- 
dents against employed adult students 
in other systems. 

A big part of what we’re doing with 
assessment, therefore, is working with 
ETS and others to address the need for 
meaningful comparisons between our 
database and those of others in tra- 
ditional settings. We need to create a 
national norming structure for adult 
students and to develop new com- 
prehensive exams that ETS can market 
elsewhere. Our critical-t hinking assess- 
ments, for example, draw a lot from 
both nationally normed exams and our 
own work. While we are beginning 
some of this collaborative work with 
ETS, I can’t guarantee that we’ll work 
with them exclusively as opposed to 
other testing services. 

How do you get your seniors to show up 
to take a three-hour test? And how do 
you guarantee that they’ll take the test 
seriously? 

These are questions that continually 
vex us. We have not overcome human 



4 — Assessment To Promote Deep Learning 



nature. We are learning that if you 
really want to pull off a post-test, it 
must be hidden from the students. We 
therefore embed it into other activities 
so they no longer know when it’s hitting 
them. It may be structured, for in- 
stance, into their last two courses, or it 
may be divided up in some other way so 
that they cannot tell whether they have 
finished, say, the critical-thinking part 
of the exam. As a consequence of em- 
bedding post-test assessments into 
coursework, the assessments become 
“high-stakes” because they’re counted 
toward the course grade. 

Another thing we have done, in or- 
der to assess continually even the post- 
test process, is to pay some students to 
take the exam. This gives us a constant 
benchmarking of students who really 
pay attention to doing the best they can 
on the exam against those who take the 
embedded exam. This is an example of 
the redundancy of our assessment sys- 
tem — purposely so because assess- 
ment is such a complex affair. 

One of the hallmarks of the American 
higher education system has been our 
lack of standardization — our free- 
wheeling, creative approach. With your 
talk of a single national exam and stan- 
dardization of the curriculum, should 
we worry about automatons teaching at 
the University of Phoenix? 

Well, all of us should always worry 
about automatons at any institution. 
But here’s my immediate response. 
Everywhere there are some faculty 
members who are really, really good. 
They do everything that we dream 
about when we think about the ideal of 
a terrific education. But we all know 
that they’re very rare. 



So at the University of Phoenix we 
have attempted to bring all our courses 
up to the level of those ideal courses. 
We’re not a faculty-centered institution; 
we’re a student-centered institution. 
Therefore, we do not want the quality of 
a student’s education to depend on the 
whim of the teacher on a particular day. 
We do not want the quality of a stu- 
dent’s education to suffer because he or 
she unfortunately can only take a 
course at a particular time on a par- 
ticular day when an idiot is teaching it, 
and when everybody knows the best 
professor is teaching the same course 
three hours later or on another day. 

We wanted to remove those luck- 
of-the-draw impediments to a quality 
education. We also don’t want to leave 
the burden of creating the curriculum 
on the shoulders of one individual pro- 
fessor, however good he or she may be. 
So we have set up a faculty governance 
structure with broad input and with as 
uniform a quality as possible. (Again, 
it’s all dependent on massive assess- 
ment and quality control processes.) 

But the struggle to create a quality 
education across an entire system does 
come with a price. Part of the price is 
that some faculty, until they get the 
hang of it, might very well feel that if 
they could have taught their course 
differently, it would have been much 
better. 

What we ask such faculty to do is 
make a formal suggestion to change the 
curriculum. If a faculty member thinks, 
for example, that we’re using a miser- 
able case study and there’s a better one, 
it is his or her responsibility to make 
that suggestion. The suggestion goes 
to campus subject matter (discipli- 
nary) curriculum committees, through 
campus-level curriculum committees, 
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all the way up to the University’s aca- 
demic council. If these groups think it’s 
a good idea, it becomes a part of the 
course. But if you are the only person 
who thinks your case is better, and ev- 
eryone else teaching it across the 
United States thinks you’re nuts, we’re 
not going to change our curriculum no 
matter how creative your approach is. 

So what you are saying then is that 
while your learning-centered, assess- 
ment-based approach gives a few less 
degrees of freedom for flexibility for fac- 
ulty, faculty can still use their discipli- 
nary expertise and deliver through a 
process by which students really learn. 

Yes. What is really uniform about the 
University of Phoenix is not so much 
the content of the curriculum as the as- 
sessment mechanisms we use to make 
sure our students achieve stated out- 
comes. As I’ve told some faculty mem- 
bers, if you can teach accounting better 
with the poetry of Emily Dickinson, do 
it. I will know whether you managed to 
pull it off or not. 

How are students assessed in their 
coursework? 

Our students are assessed continually, 
through every course, not only in terms 
of what they do in the classroom, but 
also in terms of what they do in the 
study group or section meeting that is a 
required component of every course in 
every program. 

In section meetings, students work 
in groups to develop group projects, to 
learn to interact, and to develop enough 
focus to be able to do their share of the 
group’s work. The students work on 
their own, for the most part, but the 



faculty member must be accessible to 
them. We are now beginning to create 
electronic logs that students must 
maintain as they work through the 
study group process. 

How do you view “seat time” at the Uni- 
versity of Phoenix'? 

“Seat time” is the idea that unless 
you’ve got your rear on a seat for a cer- 
tain number of hours, you’re not going 
to learn. It has its advocates despite the 
fact that absolutely nothing is put in 
place to force any assessment that 
learning actually takes place as a con- 
sequence of mere time spent on task. 

This is a topic that exercises me a 
great deal. Until very, veiy recently the 
focus of accreditation was on seat time, 
and it’s still at the heart of U.S. De- 
partment of Education thinking. Proba- 
bly no struggle between the University 
of Phoenix and regulators has been 
more intense than the issue of seat 
time. It’s somewhat ironic, because our 
students have to sit in seats longer than 
most other students do. When we say 
“an hour” of class, we mean sixty min- 
utes, not forty-five or fifty. 

The focus on the seat time issue 
thwarts all of us from providing the 
best education that we want to make 
possible. The University of Phoenix’s 
focus has been on creating a curriculum 
that is completely outcomes-based and 
on assessing those outcomes. This focus 
gives us the freedom to create the flexi- 
ble structures necessary to be able to 
provide the education our students 
want and need. 

But an outcomes-based model also 
requires us to have an assessment sys- 
tem that demonstrates that we are in- 
deed achieving what we claim we are. 
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Very high levels of assessment are nec- 
essary to address the very high levels of 
accountability that come with the re- 
moval of input-oriented notions such as 
Carnegie units or seat time. 

What about assessments at the program 
level ? 

To make sure that we don’t just assess 
student performance, we have a series 
of assessments for our programs as 
well. In addition to the comparative ex- 
ams I described earlier, on a regular 
basis we bring in reviewers from other 
universities to rigorously analyze each 
program and make sure it is structured 
to do what it’s supposed to be doing at 
the proper level of rigor. 

Libraries and information literacy are 
areas of increasing tension between you 
and accreditors. How do you measure 
outcomes in those areas ? 

We are working our way through that, 
but we are now integrating information 
literacy at the beginning and at the end 
of our programs. 

In order to give our students ac- 
cess to library resources, we have put 
together a huge online library structure 
with adequate and continually up-to- 
date resources. As they work their way 
through that huge resource structure, 
our students develop an increasing self- 
understanding of how they come to re- 
alize what it is they don’t know enough 
about. You can’t search for information 
until you realize you need it but don’t 
have it. So it’s not just information liter- 
acy but a kind of ignorance training. We 
do a lot of ignorance training, trying to 
help our students understand what 
they don’t know right now, so they can 



go out and develop the right knowledge 
base and skills. 

These s kills are absolutely critical, 
I might add, because of our 75,500 stu- 
dents, 13,500 of them are doing all of 
their degree work online. We would like 
to see more and more students shifting 
to online education, because our online 
education is better for many of our stu- 
dents than is a seminar-based envi- 
ronment. Generally, our online students 
do better than our face-to-face students 
on our exams. So information (that is, 
IT) literacy is very significant for us. 

Traditional non-profit colleges and uni- 
versities hire faculty by looking, first 
and foremost, at the credential. If you 
have it, you go through a hiring process 
and then, boom, you’re in the classroom 
to sink or swim. 

The University of Phoenix uses a 
very different process to hire faculty, 
prepare them to teach, and assess their 
performance. What systems or strategies 
do you have in place to ensure that that 
part of the puzzle has quality ? What do 
you do to recruit, train, orient, and also 
to retrain or keep your faculty ? 

The recruitment part is the result of a 
mixed bag, often people telling their 
friends and colleagues about opportuni- 
ties at the University of Phoenix. Once 
recruited, our new faculty undergo 
about five weeks of training. About one 
out of every three or so doesn’t make it 
through the process, because they really 
have to be able to teach. Not just teach, 
but teach adults who have more impor- 
tant thing s to do and have very little 
patience with incompetence. 

After initial training, our faculty 
are assessed on a class-by-class basis. 
They are then periodically assessed at 
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approximately six-month intervals and 
then annually. These reviews are done 
by the campus director of academic af- 
fairs and also by faculty peers. If our 
faculty are not making it at some point, 
we put resources into them to try and 
get them up to speed. If they still can’t 
do it, they’re just not rehired to teach. 

We use a fairly extensive end-of- 
course student survey as part of this 
process. It doesn’t ask typical questions, 
such as whether you liked the professor 
or would recommend the course to 
someone else, because that’s not our 
main concern. The real issue is whether 
the course and the rest of the system 
worked for the student. Was the faculty 
member and the curriculum appropri- 
ate? Were the financial aid services ap- 
propriate? Were other university serv- 
ices, such as admissions and the 
registrar process, appropriate? 

At the end of eveiy class, this sur- 
vey gives us feedback not only on the 
course and the professor but also on our 
entire system. 

You can see, then, that we don’t 
view assessments of students, pro- 
grams, and faculty as separate matters. 
All of our assessments are part of a sys- 
tem with layers of continual assess- 
ments about everything that contrib- 
utes to the learning and education 
process. We have weekly, monthly, 



quarterly, and ultimately annu al re- 
ports. We have consolidated reports not 
only for the entire university but also 
for faculty member by faculty member, 
campus by campus, and program 
within a campus by program within a 
campus. This gives us a series of 
benchmarks. We can tell when a par- 
ticular program is not doing well in a 
particular campus, because we can 
compare it against the same program 
on other campuses. 

We build in a tremendous amount 
of redundancy in our assessments to 
make sure that the quality of our edu- 
cation is consistent, whether you’re tak- 
ing a course in St. Petersburg, Florida, 
in Detroit, in the Netherlands, or in San 
Jose, California. 

Was it a challenge to achieve regional 
accreditation through the North Central 
Association (NCA)? 

Although we fought tooth and nail with 
the NCA through the accreditation pro- 
cess, because of it we learned a tremen- 
dous amount about how we do what we 
do and how to come to do it better. I 
think most institutions don’t appreciate 
how much they can learn from their 
fellow institutions through the process 
of accreditation. 
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Promoting 
Deep Learning 
Through Teaching 



and 



I n order to be useful, research into 
teaching and learning needs to con- 
ceptualize not only the types of 
learning we wish to encourage but also 
the types of teaching-learning environ- 
ments that are most likely to encourage 
and support such learning. Those of us 
involved in research on student learn- 
ing believe we have a framework that 
does just that. Research in Sweden, 
Britain, and other countries has led to 
new ways of thinking about how 
teaching and assessment in higher edu- 
cation influence the quality of student 
learning. We have developed a frame- 
work of concepts and categories for 
teaching and learning that makes new 
ideas readily accessible to faculty 
members. 

Research From the Student’s 
Perspective 

Research on student learning has 
sought to portray the experiences of 



Assessment 



by Noel Entwistle 

both students and faculty. The starting 
point was to describe the main differ- 
ences in how students think about 
learning and carry out their studying. 

Research has been carried out on 
two fronts. On one front, through inter- 
views with students, we studied what 
students believe learning involves and 
how they go about tasks such as read- 
ing articles or writing essays. The in- 
terviews have generally encouraged 
students not just to report their ways of 
tackling academic tasks, but also to re- 
flect on their approaches. In analyzing 
the interview transcripts, we have used 
a rigorous procedure to establish cate- 
gories and the relationship between 
those categories, a technique that con- 
tributes to a research approach de- 
scribed as phenomenography (Marton 
1994 ). 

The second line of development in 
research on student learning has been 
to design instruments to measure these 
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concepts and so study larger groups of 
students. The Approaches to Studying 
Inventory and its more recent version 
{ASSIST) include sub-scales that cover 
the categories found from the inter- 
views. We have used factor analysis of 
the sub-scales to refine our definitions 
of the categories (e.g., Biggs 1987; 
Entwistle & Ramsden 1983). 

These two complementary ap- 
proaches have established how teaching 
and assessment influence both how 
students study and the level of under- 
standing they reach. This is what will 
be discussed here. 

Conceptions of Learning and 
Approaches to Studying 

When adults were asked, “What do you 
mean by ‘learning’?” they had very dif- 
ferent conceptions (Saljo 1979; Marton & 
Saljo 1997) that fell into a hierarchy that 
parallels the development of students’ 
thinking as identified by Perry (1970). 
Some students see learning as mainly a 
matter of memorizing and reproducing 
knowledge in ways acceptable to the 
teacher. Others see learning as a way to 
establish personal meaning, by trans- 
forming the incoming information and 
ideas in relation to their existing knowl- 
edge and experience. 

The conception of learning that 
students hold substantially affects how 
they tackle everyday academic tasks, 
and that brings us to a key concept that 
describes students’ approaches to 
learning and studying. In an investiga- 
tion of how students went about reading 
(Marton & Saljo 1976), students were 
asked to read an academic article and 
were told that they would be asked 
questions on it afterwards. It became 
clear that students interpreted this in- 
struction very differently, and their 



ability to answer questions about the 
meaning of the text depended on how 
they decided to tackle the task. Some 
students sought a thorough under- 
standing of the author’s message, while 
others relied on “question-spotting” — 
learning just those pieces of inf ormation 
expected to come up in the test. 

This distinction was gradually re- 
fined to produce the concept of deep and 
surface approaches to learning. 

In the deep approach, the student 
intends to understand ideas for himself 
or herself. Such a student learns by ac- 
tively transforming. Deep learners: 

• Relate ideas to previous knowledge 
and experience; 

• Look for patterns and underlying 
principles; 

• Check evidence and relate it to 
conclusions; 

• Examine logic and argument cau- 
tiously and critically; 

• Are aware of the understanding 
that develops while learning; and 

• Become actively interested in the 
course content. 

In the surface approach, the stu- 
dent intends merely to cope with course 
requirements in a minimalist fashion. 
Such a student learns by passively re- 
producing. Surface learners: 

• Treat the course as unrelated bits 
of knowledge; 

• Memorize facts and cany out pro- 
cedures routinely; 

• Find difficulty in making sense of 
new ideas presented; 

• See little value or meaning in ei- 
ther courses or tasks; 

• Study without reflecting on either 
purpose or strategy; and 

• Feel undue pressure and worry 
about work. 

These two approaches to learning can 



ERIC 



1 0 — Assessment To Promote Deep Learning 



17 



be illustrated through the contrasting 
responses of two engineering students 
asked about their ways of studying: 

Interviewer: Tell me something 
about how are you tackling this 
course and how you work on the 
problem sheets you are given . 

Surface Approach: I suppose Tm 
mainly concerned about being able 
to remember all the important facts 
and theories that we've been given 
in the lectures . We are given an aw- 
ful lot of stuff to learn , so I just 
plough through it as best I can . I 
try to take it all down in the lec- 
tures , and then go over it until Tm 
sure they won't catch me out in 
the exams. . . . [With the problem 
sheets ], the first step is to decide 
which part of the lecture course the 
problem comes from. Then I look 
through my notes until I find an 
example that looks similar, and I 
try it out. Basically , I just apply the 
formula and see if it works. If it 
doesn't , I look for another example, 
and try a different formula. Usu- 
ally it's fairly obvious which for- 
mula fits the problem, but some- 
times it doesn't seem to work out, 
and then I'm really stuck. 

Deep Approach: It is not easy, 
you know. There is a great deal to 
cover, and I am not satisfied unless 
I really understand what we're 
given. I take quite full notes, but af- 
terwards I go through them and 
check on things that I'm not clear 
about. I find that working through 
the problem sheets we're given is a 
good way to test whether I know 
how to apply the theory covered in 
lectures , and I do that regularly. 
Once you realize what lies behind 



the problems — that's the physics 
of it and what makes it a problem 
— then you can do them. You get a 
kick out of it too , when it all begins 
to make sense. Applying the right 
formula is not difficult, once you 
know you are on the right lines. 

Interviews on studying have also 
drawn attention to the pervasive influ- 
ence of assessment procedures on 
learning and studying. Research has 
identified a third category — the stra- 
tegic approach — in which the student 
intends to achieve the highest possible 
grades. Such a student learns by reflec- 
tively organizing. Strategic learners: 

• Put consistent effort into studying; 

• Manage time and effort effectively; 

• Find the right conditions and ma- 
terials for studying; 

• Monitor the effectiveness of ways 
of studying; 

• Are alert to assessment require- 
ments and criteria; and 

• Gear work to the perceived prefer- 
ences of the teacher. 

The strategic approach can be seen in 
the comments of this student (Miller & 
Parlett 1974): 

I play the examination game. The 
examiners play it, so we play it too. 

. . . The technique involves knowing 
what's going to be in the exam and 
how it's going to be marked. You 
can acquire these techniques from 
sitting in a lecturer's class, getting 
ideas from his point of view, the 
form of his notes, and the books he 
has written — and this is separate 
to picking up the actual work 
content. 

This quotation suggests a student 
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concerned with both academic content 
and the demands of the assessment 
system. The interest in the content is 
typical of a deep approach, but the 
alertness to assessment requirements is 
strategic. 

Subsequent research using the Ap- 
proaches to Studying Inventory and 
ASSIST has found that these three 
broad factors can be further reduced into 
a single description of an idealized ap- 
proach to studying that combines deep 
and strategic and excludes any elements 
of the surface apathetic (Entwistle, 
McCune & Walker 2000). Several stud- 
ies have confirmed that the deep strate- 
gic approach leads to higher grades, but 
only where the assessment actually re- 
quires understanding to be demon- 
strated (see Entwistle 2000). 

There are important caveats, how- 
ever, in using these categories: 

1. They can only be used to describe 
the relative prominence of each ap- 
proach to studying in any student. 

It is wrong to try to put any student 
wholly into any one category. 

2. The categories are broad, indicative 
labels that do not do justice to the 
complex individual ways that stu- 
dents study (McCune 1998; Entwis- 
tle, McCune & Walker 2000). 

3. The processes needed to develop 
deep learning will vary between 
subject areas. Because approaches 
depend on context and the content 
and only partly reflect a habitual 
way of studying, an approach can 
be applied with any confidence only 
to a particular course, or even a 
specific occasion. 

Outcomes of Learning 

There seems to be widespread agree- 
ment among most higher education 



practitioners that one of our m ain aims 
is to encourage the development of 
complex conceptual understanding, par- 
ticularly what has been described as 
“critical thinking’’ (Entwistle 1997). 

Students do not develop such un- 
derstanding quickly or easily, so this 
type of thinking is only acquired gradu- 
ally over the student’s academic career. 
If we are to help students recognize the 
importance of developing complex con- 
ceptual understanding, we must be able 
to identify, and reward, different levels 
of understanding. 

Through interviews with final-year 
students at Edinburgh, we have been 
able to build on earlier research (sum- 
marized in Biggs 1999) to identify five 
levels of understanding that students 
reach as outcomes of learning (Entwis- 
tle 1995; Entwistle & Entwistle 1997; 
Entwistle 2000): 

• Mentioning: incoherent bits of in- 
formation without any obvious 
structure; 

• Describing: brief descriptions of 
topics derived mainly from mate- 
rial provided; 

• Relating: an outline with personal 
explanations, lacking in detail or 
supporting arguments; 

• Explaining: relevant evidence used 
to develop structured, independent 
arguments; and 

• Conceiving: individual conceptions 
of topics developed through 
reflection. 

Other analyses of the Edinburgh 
interviews have explored students’ ex- 
periences in reaching deep levels of un- 
derstanding. Students find the expe- 
rience is emotionally satisfying. The 
understanding they construct carries a 
feeling of “provisional wholeness” — 
complete for the time being, but still to 
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be developed further. Typical student 
comments : have yielded the following 
composite description of the experience 
of understanding: 

Understanding is the interconnec- 
tion of lots of disparate things . . . 
the feeling that you understand 
how the whole thing is connected 
up — you can make sense of it in- 
ternally. ... If I don’t understand, 
it’s just everything floating about 
and you can’t quite get everything 
into place — like jigsaw pieces, you 
know, suddenly connect and you 
can see the whole picture. . . . But 
there is always the feeling you can 
add more and more and more. . . . 
[Understanding], well, for me, it’s 
when I .. . could explain it so that I 
felt satisfied with the explanation. 

. . . [When you understand like 
that] you can’t not understand 
it [afterwards]. You can’t ‘de- 
understand’ it! 

Several students in our study ex- 
perienced this sense of connection visu- 
ally, through their revision notes. This 
suggests that they were conscious of the 
structure of their understanding and 
could review it, re-orient it, and use it to 
pull in supportive details believed to be 
“stored separately.” Their sense of 
structure gave them a logical pathway 
to guide the emerging structure of an 
essay. They could adapt this pathway 
as it developed to match the require- 
ments of the question (Entwistle 
1998a). 

We have described this direct ex- 
perience of understanding as a knowl- 
edge object (Entwistle & Marton 1994). 
Once firmly established, it seems to be 
resilient and potentially long-lasting in 



the memory, although retrieval may 
depend on the availability of strong 
cues. 

The idea of a knowledge object and 
its functions can be illustrated through 
two quotations from students on how 
they prepare for and take essay exami- 
nations (Entwistle 1995; Entwistle & 
Entwistle 1997). The visual aspect 
comes out clearly in the first quota- 
tion, while the second shows the way 
in which the student monitored the 
process. 

First Student: I can see that vir- 
tually as a picture, and I can re- 
view it, and bring in more facts 
about each part. . . . Looking at a 
particular part of the diagram sort 
of triggers off other thoughts. I find 
schematics, in flow diagrams and 
the like, very useful because a 
schematic acts a bit like a syllabus; 
it tells you what you should know, 
without actually telling you what it 
is. I think the facts are stored sepa- 
rately, . . . and the schematic is like 
an index, I suppose. 

Second Student: The more I have 
done exams, the more I’d liken 
them to a performance, like being 
on a stage; . . . having not so much 
to present the fact that you know a 
vast amount, but having to perform 
well with what you do know — sort 
of playing to the gallery. ... I was 
very conscious of being outside 
what I was writing. 

Two more quotations indicate the 
flexibility of the knowledge object and 
how its existence is experienced as be- 
ing almost independent. 
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[My way of preparing for the exam] 
gave you quite a f broad base from 
which to answer any question that 
came up on that topic , so you were 
used to being flexible in the way 
that you answered the question . It 
allowed you to adapt to different 
ways in which the question could 
be worded, and it also organized in 
your mind the relationships be- 
tween different aspects of, and ap- 
proaches to, a question . 

Following that logic through, it 
pulls in pictures and facts as it 
needs them . . . . Each time I de- 
scribe [a particular topic], it's likely 
to, be different . . . Well, you start 
with evolution, say, . . . and sud- 
denly you know where you re going 
next . Then, you might have a choice 
. . . to go in that direction or that 
direction . . . and follow it through 
various options it's offering . . . . 
Hopefully, you'll make the right 
choice, and so this goes to this, goes 
to this — and you've explained it to 
the level you've got to. Then, it says 
“Okay, you can go on to talk about 
further criticisms in the time you've 
got left” 

Conceptions of Teaching 

Research has suggested not only a hier- 
archy of conceptions of learning but also 
a hierarchy of conceptions of teaching. 
Researchers have asked faculty mem- 
bers to describe what they mean by 
learning” and “teaching” and to share 
their beliefs about teaching and as- 
sessment (Prosser, Trigwell & Taylor 
1994; Van Driel et al. 1997; Kember 
1998). The three main categories that 
have emerged closely parallel in un- 
derlying meaning those found by Saljo 



and by Perry for student conceptions of 
learning. 

Some faculty, those we call 
teacher-focused and content-oriented , 
talk about the importance of covering 
the syllabus and ensuring that students 
acquire the correct information and 
ideas. A second, smaller group, focusing 
on student activity , provide assignments 
designed to ensure active learning and 
help students develop effective study 
skills, but they still see learning in their 
own terms. We call the final group, 
smallest of the three, student-focused 
and learning-oriented . Such facility are 
most concerned with helping students 
to develop personal understanding and 
more sophisticated conceptions, and 
they design their teaching and assess- 
ment accordingly. 

Again, examples from the inter- 
views may help to clarify the two ex- 
treme categories: 

Teacher-Focused and Content- 
Oriented: It is my duty and re- 
sponsibility to help students de- 
velop the specific knowledge and 
skills which are needed to pass the 
examinations, although Tm fully 
aware that this might narrow the 
kind of education I'm giving the 
students . . . . I put great emphasis 
on objectives and making sure that 
I cover the syllabus thoroughly . In 
preparing a lecture, . . . I know ex- 
actly what notes I want the stu- 
dents to get. Students don't have to 
decide when to take notes: I dictate 
them. 

Student-Focused and Learning- 
Oriented: Tm aware of how much 
I used to assume . I now try to take 
nothing for granted and to question 
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my assumptions about how stu- 
dents learn things. . . . What I want 
to achieve is confronting students 
with their preconceived ideas about 
the subject. . . . [Conceptual under- 
standing is developed] by arguing 
about things and trying to apply 
ideas. . . . What we’re trying to do is 
... to shift [students] from the lay- 
person’s view, to what we would 
call a scientific . . . [or academic] 
view. 

Teachers with these contrasting 
conceptions of teaching tend to hold cor- 
responding views on their students and 
assessment procedures. Teacher- 
focused faculty are likely to see assess- 
ment as designed to demonstrate de- 
tailed factual knowledge of the syllabus. 
They tend to consider learning out- 
comes as being almost entirely the re- 
sponsibility of the students themselves, 
depending on ability and motivation. 
Student-focused faculty, on the other 
hand, tend to use more varied methods 
of assessment and to be aware of 
their own responsibility for encouraging 
students to develop deep levels of 
understanding. 

Promoting Deep Learning 

A wide range of studies over the last 
twenty-five years have established how 
teaching, assessment, and other aspects 
of the learning environment affect stu- 
dents’ approaches to studying and thus 
the quality of learning outcomes. 

Seven aspects of lecturing have of- 
ten been described as “good teaching.” 
Four of them — clarity, level, pace, and 
structure — are all important in con- 
veying information. Clarity describes 
audibility and visibility but has also 



been used to describe a more general 
quality in teaching. Level indicates that 
the lecturer is aware of the students’ 
current level of understanding and de- 
signs his or her teaching to match it; 
structure relates to the way in which 
the content of the lecture has been 
organized to bring out its logical 
framework. 

The other three aspects — expla- 
nation, enthusiasm, and empathy — 
seem to have the strongest effect on 
deep learning (Entwistle 1998b, 2000). 
The quality of the explanation deter- 
mines how easy it is for students to un- 
derstand the content; enthusiasm 
arouses interest and motivates learn- 
ing. Empathy describes the emotional 
climate the teacher develops, which 
powerfully affects the willingness with 
which students engage with the ideas 
presented. 

While we have a clear idea of how 
to influence levels of understanding, 
there are still great difficulties in 
achieving this ideal in practice (see 
Hounsell 1997). Linking the ideas of 
Eizenberg (1988), Wiske (1998), and 
more general literature on student 
learning, we reach the following 
conclusions: 

First, deep learning can be promoted 
through curriculum design by 

• Identifying generative, open topics; 

• Using aims to emphasize under- 
standing; 

• Incorporating authentic, relevant 
topics; 

• Defining “essential” information; 
and 

• Selecting appropriate textbooks. 

Second, deep learning can be promoted 
through teaching by 
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• Analyzing the derivation of new 
terms; 

• Emphasizing principles and 
concepts; 

• Conveying information effectively 
(through clarity, level, pace, and 
structure); and 

• Evoking a deep response (through 
explanation, enthusiasm, and 
empathy). 

Finally, deep lea rnin g can be promoted 
through assessment by 

• Focusing on understanding per- 
formance, using tasks to develop 
and demonstrate understanding 
and feedback to clarify and stress 
understanding; 

• Using techniques to tap under- 
standing, including more open- 
ended questions and less reliance 
on multiple-choice questions; and 

• Grading in relation to levels of un- 
derstanding, using qualitative cri- 
teria to boost validity. 

The infl uence of assessment on 
deep learning is clear-cut. Assessment 
techniques that encourage students to 
think for themselves — such as essay 
questions, applications to new contexts, 
and problem-based questions — all 
shift students toward a deep approach. 
Assessment perceived by students as 
requiring no more than the accurate re- 
production of information lets students 
rely on a surface approach. 

Multiple-choice questions and 
short-answer questions are the worst 
offenders (Thomas & Bain 1984; Scoul- 
ler 1998). Although multiple-choice 
questions can, of course, be written to 
test understanding, the vast majority of 
tests written by faculty require mainly 
factual knowledge (Milton, Pollio & Ei- 



son 1982). Gardiner (1994) concluded 
that most faculty members do not have 
the expertise to develop multiple-choice 
questions that test higher-level think- 
ing skills, and yet this technique is 
widely used to cope with large under- 
graduate classes. With our emphasis on 
ease and accuracy of marking, we seem 
to have lost sight of the way assessment 
controls the form of learning that stu- 
dents undertake (Scouller 1998). 

This draws attention to the influ- 
ence of students’ perceptions on their 
learning. It is not the teaching-learning 
environment itself that determines ap- 
proaches to studying, but rather what 
students believe to be required. Those 
perceptions come from the comments of 
faculty and teaching assistants, during 
their teaching and when marking term 
papers, from previous tests or exami- 
nation papers, and also from discus- 
sions with other students. 

There are also wide differences in 
how students interpret and make use of 
their learning environment (Meyer 
1991). Students with a deep strategic 
approach are much more likely to rec- 
ognize and make use of the opportuni- 
ties for deep learning provided within 
the learning environment. Less well- 
prepared students, who need the most 
support, seem less able to recognize or 
use the support that is offered (Ver- 
munt & Meyer 2000). 

Learning outcomes are affected not 
only by student approaches to learning 
and facility approaches to teaching and 
assessment but also by department and 
institutional policies and procedures 
(Entwistle 1998b), including depart- 
mental teaching ethos, course design 
and objectives, feedback to students, 
assessment procedures, freedom of 
choice, workload, study skill support, 
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learning materials, and library provi- 
sions. The interaction among these 
three elements — students, faculty, 
and organizational environment — is 
complex. 

Conclusion 

Depth of learning is affected by the cog- 
nitive processes students use. These 
processes are, in turn, affected by stu- 
dent conceptions of learning (what stu- 
dents believe learning requires of them) 
and their approaches to studying (stra- 
tegic or apathetic). Only by using ap- 
propriate cognitive processes can a deep 
level of understanding be achieved. 

But such understanding does not 
solely depend on the student. Faculty 
also differ in their conception of learn- 
ing, which affects their teaching and 
assessment methods and their attitudes 
toward students. The way in which 
faculty set up and operate the teaching- 
learning environment affects their stu- 
dents’ balance between deep and sur- 
face approaches, and so the quality of 
learning outcomes achieved. 



The following summarizes some of 
the main ways research suggests we 
can improve deep learning. 

• Provide overarching goals, genera- 
tive topics, and clear aims. 

• Relate teaching directly to prior 
knowledge. 

• Teach so as to clarify meanings 
and arouse interest. 

• Encourage metacognitive alertness 
and self-regulation in studying. 

• Introduce formative assess- 
ments designed to develop 
understanding. 

• Develop marking criteria to de- 
scribe levels of understanding. 

• Use assessment techniques that 
encourage and reward conceptual 
understanding. 

How these guidelines are imple- 
mented depends, of course, on the sub- 
ject area, the course objectives, and the 
composition of the class. How effectively 
these principles are implemented thus 
depends on the professional judgement 
and insight of faculty members. 
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Developing a 
Learning/Teaching Style 
Assessment Model for 
Diverse Populations 



S cenario One: At a regional com- 
munity college, a non-traditional 
adult student has experienced 
discomfort in most of her classes. While 
she listens to the rapid responses of her 
classmates — traditional 18-to-22-year- 
olds — she has difficulty with the ab- 
stract nature of the course content and 
the examples the other students use. 
The non-traditional student also has 
anxieties about her readiness for col- 
lege. How can the instructor better 
meet her needs? 

Scenario Two: At a Midwestern 
university, “technology in the class- 
room” has become the new mantra. De- 
cisions about hardware and software do 
not follow a consistent pattern. High- 
performance and well-skilled students 
exhibit learning preferences that match 
the instructional styles of the univer- 
sity’s highly analytical faculty. Average 
and less-skilled students are not yet at 
the level of analytical excellence that 



by James Anderson 

would facilitate their success. Their 
technology needs are different, and they 
prefer a different mode of classroom in- 
struction, both with and without tech- 
nology. What can be done for them? 

Scenario Three: At many institu- 
tions, faculty call for a “better” student 
to be admitted, one who can perform 
better, write better, think better, is 
more motivated, etc. Even when their 
institution’s mission is to admit a broad 
range of students, such facility still de- 
scribe “quality” in terms of high-end, 
well-skilled students with whom the 
faculty are comfortable. At other insti- 
tutions with the same mission, same 
type of student, and same approach to 
quality, however, their faculty promote 
success among all inco ming students. 
At these institutions, faculty are stu- 
dent-centered in terms of their teach- 
ing. What does this really mean? 

Scenario Four: The new Director of 
Disability Services suggests to the Vice 
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President for Academic Affairs that the 
institution change ’ the focus of its 
faculty-development initiatives on ac- 
commodating students whose disability 
interferes with their ability to process 
information (via lecture, text, etc.). The 
Director’s argument is that altering 
teaching style would best serve the 
needs of such students. But faculty 
counter that they already offer ex- 
tended testing time for students who 
are disabled. Is the Director demanding 
too much? 

All these scenarios point to the 
need for us to understand learning 
styles, relate them to diverse needs, and 
design appropriate assessments for 
them. 

What are learning styles? “Learn- 
ing style” refers to the preferred man- 
ner in which an individual or group 
assimilates, organizes, and uses infor- 
mation to make sense of the world, in- 
cluding a classroom or job environment. 

Learning styles can be character- 
ized by how we prefer to learn, specifi- 
cally our preferences for: 

• The type of information we receive 
(sensory vs. intuitive); 

• How we perceive information (vis- 
ual vs. verbal); 

• How we organize information (in- 
ductive vs. deductive); 

• How we process information (ac- 
tively vs. reflectively); and 

• How we understand information 
(sequentially vs. globally). 

There are many dimensions of 
learning styles, including: 

• Reflective vs. Impulsive 

• Non-affective vs. Affective 

• Elaborative vs. Shallow (repetitive) 
processing 

• Scanning (visual) vs. Focusing 



• Field-independent vs. Field- 
sensitive 

• Analytical vs. Relational 

• Independent vs. Dependent 

• Participant vs. Avoidant 

An Example: Analytical and 
Relational Learners 

An analytical learner is able to dis- 
embed information from the total pic- 
ture and focus on details. As analytical 
learners listen to a speaker or look at 
a slide, in microseconds they process 
everything they’re absorbing against 
what they’ve already stored. As they do 
this, they dis-embed information that 
doesn’t agree with what they already 
think or need to know. 

Analytical learners think sequen- 
tially and structurally. They: 

• Can learn inanimate and imper- 
sonal material; 

• Don’t need to hear examples that 
they can literally reach out and 
touch in terms of cultural or expe- 
riential relevance; 

• Have a good memory for irrelevant 
and abstract information, such as 
calculus; 

• Are very task-oriented; they can 
persist with unstimulating tasks; 

• Do not let their performance be af- 
fected greatly by the opinions of 
others. 

This style matches up with traditional 
educational environments, and such 
students are usually very successful 
academically. 

Relational learners, at the other 
end of the continuum, prefer to look at 
information as a total picture, focusing 
on a gestalt rather than on details. 
They: 

• Lake to take it all in, because they 
need to make affective decisions 
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about the information (it’s not that 
they are emotional learners; they 
want to understand cognitive, 
analytical information like ev- 
eryone else does); 

• Think improvisationally and 
intuitively; 

• Prefer materials that have a hu- 
man, social content and cultural 
relevance. 

• Have a good memory for verbally 
presented ideas and information, 
especially if it is relevant to them; 

• Tend to be more task-oriented in 
non-academic areas; 

• Let their performance be influ- 
enced by expressions of confidence 
or doubt in their ability from 
authority figures; 

• Prefer to withdraw from unstimu- 
lating tasks. 

Relational learners have a processing 
mode that asks affective, personal, and 
social questions about information. If 
the information doesn’t make sense in 
that context, if it doesn’t come back 
with positive feedback, or if they don’t 
see relevance and familiarity in it, they 
lose interest in it. It’s not that the in- 
formation is not valuable; they just lose 
interest. 

You can see that the relational 
learning style conflicts with traditional 
school environments, especially in sci- 
entific or technical fields. Think of the 
math courses you’ve taken in your life. 
How many teachers opened a lecture by 
saying, “By the end of class, you will 
understand how what I’m going to talk 
about today adapts to your personal 
life”? Relational learners can’t get 
through tough, abstract bottlenecks 
such as Ohm’s Law in electrical engi- 
neering, debits in accounting, vectors 
in physics, molecules in chemistry — 



concepts for which we don’t provide 
human/social content. Faculty will often 
ignore the needs of such relational 
learners, because faculty themselves 
are more analytical and task-oriented. 

Another Example: Affective 
Learners 

Affective students care about and 
evaluate the messenger as well as the 
message. They don’t leave their affect 
outside the door. They may look at the 
instructor and say, “Boy, he sure is 
dressed funny today!” You and I would 
say, “What difference does that make?” 
but affective students focus on the so- 
cial dynamics of the classroom. They 
value the instructor’s presentation of 
self — i.e., whether the instructor im- 
mersed herself into the presentation 
and dialogue — as much as they value 
the presentation of ideas and content. 

When I taught psychology courses, 
I always used my own life as an exam- 
ple, so my students could see the incor- 
poration of self into my delivery of the 
content (except, that is, in abnormal 
psychology!). Affective learners want to 
convert what is abstract to something 
that has experiential, cultural rele- 
vancy. They like periodic feedback that 
tells them something about themselves 
as learners and connects to where they 
are as learners. 

A Third Example: Visual Learners 

Visual learners prefer to learn by scan- 
ning information from a lecture, text, or 
other source, searching for cues or focal 
points to which they can connect or 
relate. 

We all engage in visual learning 
when we cha nn el-surf late at night and 
scan channels for five seconds apiece. 
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We’re looking for visual cues that tell us 
we should stay on k particular channel 
a little longer. In only a few seconds we 
decide, based on visual information, 
whether to stay or keep on channel- 
surfing. 

Some students look at all the cues 
an instructor gives and select the most 
relevant information; others come 
primed to hear only the cues they want 
and ignore most other information. 
Some students prefer highly affective 
cues, which are examples that have 
relevancy and familiarity in terms of 
the student’s experience. 

If visual learners can’t find such 
focal points, they focus on something 
else, and things begin to break down: 
They become bored or distracted. If stu- 
dents who are both visual and affective 
learners don’t see a plethora of affective 
cues within three classes, they will time 
out the instructor. They may not come 
to class. They may ask a friend to take 
notes. We know that a lot of dysfunc- 
tional things can happen when learning 
preferences are not accommodated 
early on. 

When cues are provided for visual 
learners, however, their thinking is 
stimulated. They raise questions about 
the information. They want to pursue 
the subject more. The next step is al- 
most imperceptible: They begin to draw 
on past experience to make the learning 
experience more holistic, more of a 
gestalt. 

Why do some students need to 
read a chapter three or four times to get 
detail, and some need to read it only 
once? One reason is skill in pattern rec- 
ognition. Every textbook chapter in 
every discipline has three components: 
details about the chapter’s subject, ex- 
planations of the details, and example 



of the explanations and details. Stu- 
dents who lack pattern recognition read 
all this together and can’t distinguish 
details from explanations and exam- 
ples. If you ask them a test question 
about a detail, they may provide an ex- 
ample instead. When they find out their 
answer is wrong, they’ll ask, “Why is 
this wrong? It was in the book!” 

Some visual learners lack skill in 
pattern recognition. Those who are 
skilled in pattern recognition can ex- 
tract detail and separate it from exam- 
ple and explanation. Here’s the key: 
They do this by drawing on past experi- 
ence to develop the gestalt, a holistic 
perception of what’s going on. 

Once visual learners find cues and 
use them to create a holistic perception, 
they demonstrate goal-directed behav- 
ior: They study harder. They get in 
study groups. They raise questions in 
class. They see the instructor after 
class. When they read a chapter, 
they’re more willing to think about pat- 
tern recognition. 

Learning Styles as Continuums 

I may have given the impression that 
learning styles fall into bipolar dis- 
tributions — either students are visual 
learners or they’re not, affective learn- 
ers or they’re not. In reality, learning 
styles are on continuums. We all have a 
learning style on each continuum, just 
in different places along the line. And 
there are instruments that allow us to 
identify where students are on these 
continuums. 

Which learning styles are most ef- 
fective? We have determined from re- 
search that students who are reflective, 
non-affective, elaborative-processing, 
scanning, field-independent, analytical 
learners are highly successful in both 
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two-year and four-year colleges. They 
are our dream students. If they also 
come with a 1450 SAT and several Ad- 
vanced Placement courses, their in- 
structor can walk in every day and say 
anything, and they’re going to get it. 
But in the real world, we want all our 
students to succeed, not just those 
primed for success. 

An Evolving Discussion 

The discussion of learning styles has 
not moved with the same impetus that 
many other discussions in higher edu- 
cation have. The discussion of learning 
communities, for example, has moved 
from a conceptual model to an imple- 
mentation model; and discussions of 
teaching have moved to discussions of 
the scholarship of teaching. We have 
moved at warp speed in developing 
models for classroom assessment; but, 
again, we have not followed suit in de- 
veloping assessment models that ac- 
commodate learning styles. 

Why has the evolution of our ideas 
about learning styles moved at such a 
slow pace? I offer four reasons. 

The first is that our conceptual 
models of learning styles have become 
locked into the places where they origi- 
nated as research topics: cognitive psy- 
chology, visual perception, etc. Because 
they are locked in as research topics, 
they have not yet been applied signifi- 
cantly to teaching and lea rning at a 
practical, performance-based student 
level. 

The second reason that our concep- 
tual models of learning styles have not 
evolved is because we haven’t connected 
them to classroom performance, writ- 
ing, thinking, student success indica- 
tors, retention models, and so on. These 
connections do show up in the litera- 



ture, but they’re not an integral part of 
our dialogue. 

The third reason is because many 
campuses are not yet student-centered. 
If a campus is lax in merely thi nkin g 
about what good teaching is, and if it 
lacks a student-centered approach to 
teaching and learning, why would it 
want to examine student learning pref- 
erences, learning strategies, and 
learning styles? 

The final reason I offer (and mine 
is not an exhaustive list) is because to 
discuss learning styles in earnest is ul- 
timately to discuss differential per- 
formance of certain groups and the re- 
lationship between traditional teaching 
styles and the learning styles of diverse 
populations. Now we’ve moved from a 
research discussion to a political dis- 
cussion. And if faculty don’t want to ad- 
dress student-centered teaching, why 
would they want to address the politics 
of teaching and its connection to differ- 
ent groups? 

Advancing the Conversation on 
the Needs of Diverse Learners 

We all accept the notion that when we 
teach or engage in any type of academic 
support, not all our students have the 
same needs. All students have diverse 
needs that we want to meet. We want 
them to be better thinkers, better writ- 
ers, better problem solvers, and so on. 

Then there are particular groups of 
students who have unique needs that 
we also want to meet. Consider a re- 
turning adult who is less-skilled, rais- 
ing a family and working full-time. If, 
because of poor advising, this student 
gets a full load of tough courses with 
abstract course content, he has a 100 
percent chance of getting a D or F in 
almost every one of those courses. 
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So why haven’t we looked out for 
the unique needs ! of certain diverse 
learners? One reason that diversity is 
not considered in many campus dis- 
cussions of teaching, learning, assess- 
ment, scholarship, and research is that 
we ignore its natural fit with these en- 
deavors. To see how easily diversity can 
be incorporated into discussions of 
learning style, recall how visual learn- 
ers learn best: by drawing on personal, 
social, and cultural experiences to make 
the learning experience more holistic. 
By noting this, we have introduced di- 
versity into the conversation on learn- 
ing styles. 

Cultural Differences in Learning 
Styles 

When I began to study learning styles, 
the first thing I discovered was that 
there was no literature on group differ- 
ences, with the exception of some work 
on differences in such things as visual 
perception, field independence, and 
field sensitivity. There were also some 
comparisons of Eurocentric groups with 
other groups — for example, Scottish 
children with Zambian children — but 
nothing connected to classroom per- 
formance, higher-order s kills , etc. 

But today we know a lot more. Do 
certain racial, ethnic, or cultural groups 
lean more toward some ends of the con- 
tinuums than others do? Yes. Dif- 
ferences in learning styles are so pro- 
nounced that we can make clear dis- 
tinctions among cultural groups, racial 
groups, gender groups, age groups, and 
so on. Students from certain groups 
tend to be disproportionately relational, 
affective learners. In the late 1980s, I 
leaned toward thinking there was 
something called a “Black Learning 
Style” or “Women’s Lea rning Style.” 



But I’ve changed since then. If I 
could select two factors that probably 
have the most impact on students’ 
learning styles and group differences, it 
would be class and prior educational 
experiences, be they in the family or in 
school. If you map the learning styles of 
whites in Appalachia and blacks in 
Mississippi, they'll look exactly alike. If 
you map the learning styles of students 
of color at Reed College in Oregon and 
at Harvard, again they’ll map in similar 
ways . . . that is, bright, analytical stu- 
dents, regardless of race, will show up 
that way. 

Some researchers are beginning to 
focus on a broader approach that iden- 
tifies other dimensions of learning 
styles. Madge Willis (1989), for exam- 
ple, t alks about learning styles of Afri- 
can-American children: 

• Social / Affective: They tend to be 
people-oriented and emphasize the 
affective domain. Social interaction 
is crucial, and social learning is 
common. 

• Harmonious: They tend to respect 
and encourage the interdepend- 
ence and harmonic/communal as- 
pects of people and environment. 
They seek knowledge for practical, 
utilitarian, and relevant purposes. 
They seek synthesis and holistic 
approaches to experiences. 

• Expressive Creativity: They tend to 
be creative, adaptive, variable, 
novel, stylistic, and intuitive. They 
prefer simultaneous stimulation 
of multiple senses and oral 
expression. 

• Non-Verbal: Non-verbal communi- 
cation (intonation, body language, 
movement, and rhythm) are vital 
to helping these students learn. 

Willis found that the response of di- 
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verse students in a classroom is affected 
by the following: 

• They care about and evaluate the 
messenger (the instructor as well 
as the message). They value the 
instructor’s presentation of self, as 
well as the presentation of ideas 
and content. 

• They are concerned about the so- 
cial dynamics of the class. 

• They want abstract concepts con- 
verted into something that has ex- 
periential and/or cultural rele- 
vance or familiarity. 

• They like periodic feedback that is 
aimed at the learner rather than 
the error, connects to the learner’s 
developmental level, and focuses 
on what they can learn now. 

Should We Encourage Students to 
Change Their Learning Style? 

Students come to us with learning 
styles developed over many years, 
rooted in their culture, family back- 
ground, and prior educational experi- 
ences. Do we want to go as far as I’ve 
suggested in studying group differences 
in learning styles? In doing so, are we 
suggesting that some groups are defi- 
cient in learning style? Of course not. 
We don’t make evaluative judgments 
about learning styles; we affirm learn- 
ing styles as a reflection of a student’s 
heritage. 

But should we nonetheless encour- 
age some students to modify their 
learning style? Yes, because we live in 
the real world. The students who will 
be most successful in college move from 
the affective toward the analytical side. 
I look at the performance of affective 
students in tough courses, especially 
when they are dragging their affect into 
the classroom when it doesn’t need to 



be there — i.e., they can express their 
affect anywhere else, but in that class- 
room they need to be very focused. 

It is not necessarily difficult to 
modify one’s learning style. Most of us 
can move up and down these continu- 
ums, and we know exactly when we 
should do so. When we’re in a restau- 
rant with Mends, for example, there’s 
no need to have a highly analytical dis- 
cussion about the caloric breakdown of 
everything that’s on our plate or about 
the class differences among the people 
sitting around us. But if you’re giving a 
conference presentation, you’re going to 
move toward the analytical end of the 
continuum because you’re addressing 
high-end, well-heeled learners. 

We all have the jobs we have be- 
cause we’re good at this higher educa- 
tion thing. If there’s one thing that we 
can share with students and help them 
learn, it’s how to move up and down 
these continuums. 

What about students who are 
highly analytical and devoid of affect? 
Don’t they need to move to the affective 
side? Yes, they do at some point in life, 
but not necessarily while they’re getting 
through college courses. 

The Importance of Framing 
Questions 

What are the implications of all this for 
each of us? If you want to develop an 
assessment model that addresses the 
needs of diverse populations, the most 
important thing that you can do is 
frame the questions that you want an- 
swered. What do you want to know? 
Why? 

Here are some examples of fram- 
ing questions to ask yourself: 

• Do we seek only to identify stu- 
dents’ attitudinal dispositions to- 
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ward learning, without connecting 
them to teaching? 

Do.we only want to know how stu- 
dents feel about learning and their 
learning preferences? 

Does our faculty have an under- 
standing of general learning 
styles? (If we’re at an open admis- 
sions institution, how can they not 
have an understanding, since 
every day in every class there is 
such a wide range of learners sit- 
ting in front of them across differ- 
ent dimensions, skill levels, 
learning styles, learning prefer- 
ences, and motivational levels?) 
Who impacts faculty perceptions of 
student learning styles? 

Do we want to assess our students’ 
learning styles? Whose responsi- 
bility is it to do this and get the in- 
formation to faculty so they under- 
stand their students’ needs? 

If we do collect such information, 
so what? What do we want to do 
with it? 

For those whom we identify as 
being most at risk for success, 
based on valid, objective informa- 
tion, what does it mean? What are 
we doing about it? 

How can we move less successful 
students incrementally through a 
process that allows them to be- 
come more successful? How do we 
move students from being affective 
to being more analytical? 

Should we adapt instructional 
styles to accommodate learning 
styles? What does that mean 
for faculty development? What 
does that mean for classroom 
assessment? 

What cognitive, affective, and cul- 
tural assets do diverse students 



bring to learning environments, 
and how do these assets facilitate 
or inhibit their performance? How 
can we tap into those assets? 

• How do all aspects of diversity fit 
into assessing performance? 

• To what degree do we want to as- 
sign students to sections according 
to styles and then match them up 
with certain instructors? Why 
should students who are less- 
skilled and affective learners be 
placed with a highly analytical 
instructor? 

• Should we use information about 
learning styles to help students 
decide what to emphasize or de- 
emphasize in their studies? Stu- 
dents who take an entire semester 
online and who are highly verbal 
and less visual in learning prefer- 
ence may experience difficulty. 

Can a student who’s a highly affec- 
tive, relational learner succeed at a 
Research I institution, where most 
of the faculty she encounters in 
mathematics, science, and techni- 
cal areas will teach in an analyti- 
cal manner? 

• Should the relationship of learning 
style research to educational out- 
comes affect what we do with fac- 
ulty development? 

• What new assessment methods 
are needed? 

• Where’s the next frontier to help 
us accurately evaluate and portray 
learning styles as they are im- 
pacted by diversity? 

Once we’ve considered these kinds 
of questions, we can begin to think 
about reasons for not only doing more 
learning style assessment but also in- 
corporating diversity. We may want to 
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give students a learning styles-pref- 
erence survey simply to give self-assess- 
ment feedback to students, so they can 
see themselves, maybe for the first time 
ever, as a learner. 

We can go from there to doing co- 
hort comparisons, looking at clusters of 
behaviors that we see in groups. For 
example: What clusters of behaviors are 
associated with success or failure in be- 
ginning science or math courses? What 
clusters of behaviors are associated 
with success or failure across what 
groups in engineering? 

Next Steps 

We are slowly moving from a generic 
model of learning style to a more com- 
prehensive model that incorporates the 
diverse needs of all students and the 
unique needs of diverse groups, includ- 
ing their learning styles. 

There is currently no effective as- 
sessment of learning styles and diver- 
sity that will enlighten us in significant 
ways about student performance, stu- 
dent success, student learning, etc. Tm 
working on an instrument that has 
been pilot-tested for reliability and va- 
lidity at five institutions, and we will 
soon be pilot-testing it at five more. 
We’re laying to develop an instrument 
that profiles generic learning styles and 
also correlates that information with 
other critical dimensions, such as stu- 
dent-student interaction and student- 
instructor interaction. 

Given limited resources, what 
steps can institutions take to better ad- 
dress the needs of diverse learners? Be- 
gin by developing a strong teaching ini- 
tiative around a more general area and 
then incorporate attention to diverse 
learning styles into it. At North Caro- 
lina State, a group called the Hewlett 



Fellows focuses on inquiry-guided 
teaching and promoting active learning. 
Faculty are very enthused about it. But, 
if we had first tried to develop a learn- 
ing style initiative focusing on effective 
teaching, I predict it wouldn’t have been 
as successful. 

Another possibility is to create co- 
operative clusters or learning com- 
munities, provided that they are de- 
signed to accommodate diverse groups. 
Diversity is not as present as it should 
be in learning community research. 
There’s an inherent assumption that 
learning co mm u ni ties automatically ac- 
count for diversity, but that’s not true. 
For example, if you set up voluntary 
curricular learning communities, di- 
verse students will not necessarily sign 
up. They do not see the inherent value 
of clustering across courses. 

Cooperative clusters show prom- 
ise, however. Sheila Tobias (1992, 1990) 
has studied cooperative clusters associ- 
ated with the success of women in sci- 
ence and mathematics. Uri Triesman 
(1992) has done the same with under- 
represented students and students of 
color, especially in mathematics. 

But cooperative learning models 
don’t attract everyone. A student who is 
introverted and less-skilled and doesn’t 
understand the culture of college is not 
going to be assertive in cooperative 
learning approaches. That student will 
not participate actively in lea rning 
communities, and that student will be 
silent in chat rooms. 

It’s a challenge to address the 
needs of diverse learners, because it’s so 
difficult to reallocate resources from 
things that aren’t really significant and 
don’t yield outcomes of consequence. 
But we should do facility development 
on this subject, and we should have 
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them in? They’ll just continue having 
problems. These are two areas in which 
we should all invest resources. 



something for students coming into our 
institutions who historically have been 
identified as having the most problems. 

If we don’t do that, why keep bringing 
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A Few Good Measures: 
The Impossible Dream? 



L et me begin by noting that I’m an 
institutional researcher. As you 
may know, institutional re- 
searchers have all the answers, but we 
don’t, unfortunately, have the ques- 
tions. So I’m going to take you on a 
journey to figure out what the questions 
are, rather than what the answers are. 

Since I’ve worked with Trudy 
Banta for eight years or so, I know that 
one always starts with a purpose. So 
that’s where we’ll begin: My major pur- 
pose is to help you determine whether it 
really is possible to have a few good 
measures. What will the answer be: yes 
or no? Well, it depends — on what you 
mean by “few,” on what you mean by 
“good,” and, as President Clinton en- 
lightened us, on what you mean by “is.” 
And if by chance we figure out that 
there aren’t a few good measures, then 
how, where, when, and in what context 
can performance measures be useful? 
My second purpose is to address this 
question. 

My third purpose is to explore 
some organizing concepts that may be 
useful in developing performance meas- 
ures. My final purpose is to help you re- 
alize that no one else has figured all 
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this out. Lately, when I attend confer- 
ence sessions on performance indica- 
tors, I worry that someone else has it 
figured out. I have yet to have my wor- 
ries confirmed. 

What Is a Performance Indicator? 

Performance indicators — or, as I’ll use 
the term interchangeably, performance 
measures — mean many things to dif- 
ferent people. At a recent site visit for 
an evaluation project in which I’m in- 
volved, Peter Ewell characterized per- 
formance measures as a way to tell 
someone something about what you do. 
Peter’s simple characterization stuck in 
my mind. Fd like to add slightly to this 
simple definition: Good performance 
measures also tell something about how 
well you do it, whether it works, and 
what your plans are for improvement. 

Most people also expect perform- 
ance indicators will be quantitative, 
with a specific statistic, number, or 
perhaps a trend line that shows what 
and how you’ve been doing. 

For many people, performance in- 
dicators are something that gets done to 
you rather than something you do, 
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somewhat like taxes or a bad cold. But, 
because the best defense is a good of- 
fense, I’ll focus my comments on how 
you can be involved in developing per- 
formance measures proactively, rather 
than reacting to external mandates. 

And now, a song for inspiration (to 
the time of “Just My Imagination”): 

I look at my email, a message from way 
on high, 

The chancellor wants measures that all 
our constituents will buy, 

To please the board of trustees, state 
legislators of every sort, 

That will also improve our ranking in 
U.S. News & World Report. 

I whip out my palm top remote access to 
the PeopleSoft Data Mart, 

Download value-added learning 
outcomes, and that was just a start. 
Faculty workload, courses taught, 
grants obtained and awards, 

The impact of our civic engagement on 
communities that we support. 

But it was just my imagination running 
away with me, 

It was just my imagination, running 
away with me. 

Soon, there’ll be measures, on which we 
can all agree, 

All our constituents and stakeholders 
will clearly see, 

The faculty do more than just work for 
themselves, 

And institutional researchers write 
reports that don’t just sit on shelves. 

But it was just my imagination running 
away with me, 

It was just my imagination, running 
away with me. 



It All Depends on Your Perspective 

Determining which performance indi- 
cators are best for your situation de- 
pends on three factors: the unit level in 
which you work (e.g., department, 
school, institution, system), your role in 
that unit (e.g., chair, dean, provost), 
and your target audience (e.g., your 
faculty, prospective students, the state 
legislature). 

Let’s think of some potential indi- 
cators, either from the audience per- 
spective or from the provider perspec- 
tive. Suppose, for example, that your 
unit is college- or university-wide, your 
role is provost, and your audience is 
your faculty. You want a few good 
measures for internal accountability to 
your faculty. What are some possibili- 
ties? The employment ratio of gradu- 
ates, employer satisfaction with student 
performance, graduation rate by major, 
the number of people who take required 
courses at your institution as opposed 
to another college or smother provider, 
licensure pass rates ... all are candi- 
dates that might be appropriate. 

Now suppose you are provost 
again. What would you want to tell 
prospective students? Perhaps the per- 
centage of students going on to gradu- 
ate school, student employment out- 
comes, student satisfaction, the quality 
of your programs, or information about 
technology and innovation. 

Now let’s move to the audience 
side. You’re a student. What would you 
want the provost to tell you? Perhaps 
the average starting salary of recent 
graduates, opportunities for meaningful 
interaction with faculty, class size or 
the student/faculty ratio, the acceptance 
rate at professional schools, classroom 
resources, transferability of class work, 
and cost. 
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Now suppose you’re a member of 
the faculty. What would you want the 
provost to tell you about how your col- 
lege or university is doing? Perhaps 
faculty salary benchmarks, equity in 
allocating positions, quality of incoming 
students, or student satisfaction. 

Now let’s say you’re a provost 
talking to a state legislator. What do 
you want her to know about your insti- 
tution? Perhaps the graduation rate 
of first-generation students, retention 
rates, the number of students graduat- 
ing in four years, cost per degree, per- 
cent of graduates remaining in the 
state, student competency, licensing 
pass rates, grant funding, the percent- 
age of lower-division student credit 
hours taught by full-time tenured fac- 
ulty, the number of registered voters in 
the legislator’s district attending your 
college, hours of community service per 
student, and enrollment and retention 
rates of students of color. 

As you can see from these exam- 
ples, it’s not difficult to come up with 
many good measures. Our original 
question, however, was whether it is 
possible to come up with just a few. To 
answer this, I will first take you 
through a somewhat personal tour of 
my experiences in developing perform- 
ance indicators, and then consider some 
emerging trends. 

Lessons Learned From Scholarship 
and Theory 

Measurement theory posits that meas- 
ures exist only within a context of ideas, 
concepts, or theories. Measures should 
address how those ideas are manifest in 
observable behaviors, objects, or events. 
From these empirical observations, we 
can then reassess and refine our con- 
cepts and theories. We know this as the 



“inductive-deductive cycle”: conceptual- 
izing, hypothesizing, measuring, and 
generalizing in a recurrent or perhaps 
spiraling cycle. Without this cycle, 
measurement has no meaning. Peter 
Ewell and Dennis Jones (1994) boil the 
inductive-deductive cycle to a few key 
words: “Above all, think before you 
count” (34). 

In his book Conceptualization and 
Measurement in the Social Sciences, 
Blalock (1982) notes that “unless very 
careful attention is paid to one’s theo- 
retical assumptions and conceptual ap- 
paratus, no array of statistical tech- 
niques will suffice” (9). This is a fancy 
way of saying “garbage-in, garbage-out,” 
which is demonstrated by many per- 
formance measures that we now use, 
including graduation rates and funding 
per FTE. Do these measures tell us 
anything meaningful? Are they measur- 
ing the same things across different 
places? Are they comparable? (We know 
the answer: Comparability is very 

limited.) 

Trudy Banta and I co-edited a vol- 
ume of New Directions for Institutional 
Research, entitled Using Performance 
Indicators to Guide Strategic Decision 
Making (Borden & Banta 1994). I will 
now share the sound bite lessons that I 
believe emerge from each of the chap- 
ters of that volume, one of which I’ve 
already mentioned. [Chapter authors 
are indicated parenthetically.] 

• Where you stand on performance 
indicators depends on where you 
sit: your perspective and your role 
(Borden & Bottrill); 

• Think before you count (Ewell & 
Jones); 

• In Europe, performance indicators 
are “out”; quality assurance is “in” 
(Jongbloed & Westerheijden); 
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• Pis are useful, if the P stands for 
“process” (Dorris & Teeter); 

• Key performance indicators can be 
the fuel for a strategic decision en- 
gine (Dolence & Norris); and 

• Activity-based costing can yield 
performance measures relating to 
the costs of doing business (De- 
Hayes & Lovrinic). 

I’ll go into just a little more depth 
about one of these chapters to illustrate 
the kind of thinking that emerges if you 
go beyond the sound bite level of analy- 
sis, which is not common among many 
purveyors of performance indicators 
(especially the externally mandated 
kinds). 

Although we usually think of per- 
formance indicators as measures of in- 
dependent aspects of an institution’s 
performance, Michael Dolence and Don 
Norris suggest in their chapter that it is 
important to consider the interaction of 
strategic initiatives across performance 
indicators. For example, average section 
size and cost per credit hour are two po- 
tential college or university perform- 
ance indicators that are often inversely 
related: As section size decreases, costs 
typically increase and vice versa. Do- 
lence and Norris propose an interesting 
and engaging group process for exam- 
ining the impact of strategic initiatives 
across key performance indicators. As 
with all such planning processes, the 
greatest value is derived more from the 
process than from the product. 

In the concluding chapter, Trudy 
Banta and I (1994) propose five criteria 
for effective performance indicators: 

1. They start with purpose. 

2. They are aligned throughout the 
organization. 

3. They are aligned across inputs, 



processes, and outputs. 

4. They coordinate a variety of 
methods. 

5. They are used in decision making. 

Lessons From the American 
Productivity and Quality Center 

In another chapter of my life, I partici- 
pated with representatives of about 
twenty other universities in an Ameri- 
can Productivity and Quality Center 
(APQC) benchmarking study called 
“Measuring Institutional Performance 
Outcomes” (APQC 1998). 

For the benefit of those of you who 
are not familiar with APQC bench- 
marking studies, they employ a largely 
qualitative method to identify institu- 
tions, both inside and outside of higher 
education, that are exemplars in the 
target activity of the study. Through a 
series of group project meetings and site 
visits to the “best practice” institutions, 
study participants identify a set of 
emergent themes that characterizes 
best practice. 

The following themes emerged 
from the study in which I was involved: 

• The best performance measures 
communicate the institution’s core 
values. 

• Good institutional performance 
measures are carefully chosen, re- 
viewed frequently, and point to ac- 
tion to be taken on results. 

• External requirements can be ex- 
tremely useful as starting points 
for developing institutional per- 
formance measurement systems. 

• Performance measures are best 
used as “problem detectors” to 
identify areas for management at- 
tention and further exploration. 

• Clear linkages between perform- 
ance measures and resource allo- 
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cation are critical, but the best 
linkages are indirect and 
non-punitive. 

• Performance measures must be 
publicly available, visible, and con- 
sistent across the organization. 

• Performance measures are best 
considered in the context of a 
wider transformation of organiza- 
tional culture. 

• Organizational cultures supportive 
of performance measures take 
time to develop, require consider- 
able “socialization” of the organiza- 
tion’s members, and are enhanced 
by stable leadership. 

• Performance measures change the 
role of managers and the ways in 
which they manage. 

• You cannot ‘lead” with perform- 
ance measures. 

• Performance measures emerge 
from a broader culture of evidence 
— that is, they are part of some- 
thing bigger. 

But the major conclusion of the APQC 
study, in my estimation, is that there is 
probably no single set of a few good 
measures, which answers our first 
question. 

Having said this, I will admit that 
one of the non— higher education institu- 
tions identified as a best practice part- 
ner in the APQC study did have a sin- 
gle set of five performance measures. 
However, these five measures belie 
far more extensive information-based 
analyses, measures, and evaluation 
processes that are going on underneath 
the surface. 

Lessons From the IUPUI Experience 

During my eight-year tenure at Indiana 
University Purdue University Indian- 
apolis, the university has undertaken 



three approaches to developing and 
using performance indicators. 

I call the first one the “laundry 
list” approach. We started with the five 
major strategic themes described in our 
mission statement: student learning, 
responsibilities for excellence, centrality 
and community connections, collabora- 
tion, and accountability and best prac- 
tices. Each of these themes was further 
elaborated by three to eight institu- 
tional goals, for a total of twenty-nine 
institutional goals. 

The twenty-nine institutional goals 
were not exactly concretely measurable 
concepts. For example, one of the goals 
within the theme “Responsibilities for 
Excellence” was “Having expert faculty 
in all our disciplines whose teaching 
and scholarly work are respected and 
celebrated within and outside the insti- 
tution.” What measure might we use to 
assess whether we have attained this 
goal? Obvious possibilities include the 
number of rewards and recognition at- 
tained by our faculty, their presenta- 
tions, publications, editorial board 
memberships, and so on. 

I’ve just quickly rattled off four 
concrete measures for one of the 
twenty-nine institutional objectives 
specifically mentioned in our strategic 
plan. If that is an average number, then 
we are now up to about 120 indicators 
total. 

Then consider the questions we 
would have to resolve in order to meas- 
ure just one of these indicators: number 
of articles published in refereed jour- 
nals. First, is that an adequate defini- 
tion? Or do we need to consider the 
“quality” of each journal? Second, have 
you ever tried to collect reliable infor- 
mation from a faculty body numbering 
more than 1,500 on any aspect of 
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faculty scholarship? 

As you might imagine, this 
“laundry list” approach died of its own 
weight very quickly. 

I call IUPUI’s second approach the 
“performance report” approach. Under 
this approach each school prepares an 
annual report of its activities — what 
was done well, and what measures 
were used to make that determination. 
The central planning support staff re- 
view these documents in search of ex- 
amples to include in an annual campus- 
wide summary of accomplishments re- 
lating to the institution’s strategic 
themes, goals, and objectives. We (i.e., 
the central planning support staff) then 
develop some quantitative summaries 
that relate to the general theme areas 
(e.g., student learning), without wor- 
rying about covering every specific goal 
and objective. Finally, we include an 
appendix of other general quantitative 
trends related to students, faculty, staff, 
and budgets. 

The first year we prepared this re- 
port, we noticed how our available 
measures were imbalanced relative to 
IUPUI’s strategic themes. For example, 
we have a lot of data on student enroll- 
ment, but not as much on student 
learning or on centrality and commu- 
nity connections. 

This approach nonetheless pro- 
duces a report that is closely aligned 
with real institutional planning, report- 
ing, and improvement efforts. If we im- 
prove our planning, reporting, and im- 
provement efforts, the annual per- 
formance report should become better, 
perhaps even including something we 
might recognize as a set of performance 
measures. 

As an outgrowth of this approach, 
we have begun to rethink our broad- 



stroke planning and evaluation proc- 
esses, leading to IUPUI’s third ap- 
proach, which I call the “Core Mission- 
Core Indicators” approach. Realizing 
that the clarity and succinctness of our 
performance indicators are necessarily 
tied to the clarity and succinctness of 
our mission, we started by revising our 
mission statement to a much simpler 
statement: 

The VISION oflUPUI is to he rec- 
ognized as one of the best urban 
universities. For IUPUI, achieving 
the status of an outstanding urban 
university means demonstrating: 

• Effective Student Learning 

• Excellent Research and Scholar- 
ship, and 

• Exemplary Civic Engagement 

Our task now is to identify indica- 
tors within these three categories. Our 
early thinking suggests that we may be 
able to come up with far fewer indica- 
tors than the 120 or so inherent in our 
laundry fist process. 

Within the “Effective Student 
Learning” mission, we’ve identified two 
kin ds of goals: enrollment management 
and student learning. Within enroll- 
ment management, we examine the 
quality and diversity of incoming stu- 
dents, rates of student progress, stu- 
dent satisfaction, and transfer articu- 
lation and success. Within student 
learning, we examine assessment re- 
sults, licensure rates for professional 
programs, students’ self-perceptions of 
learning gains, and employer ratings of 
s kills and competencies. Assessment 
results aren’t really a performance 
measure, because you can’t boil them 
down to a single meaningful number, 
but you can use an indicator to dem- 
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onstrate that you have them. 

Within the “Excellent Research 
and Scholarship” mission, we look at 
grants and funding, awards and rec- 
ognitions, National Centers of Excel- 
lence, and the ISI database, which pro- 
vides information on citations of articles 
written by faculty. 

Within the “Exemplary Civic En- 
gagement” mission, we examine our 
contributions through education and 
research, specifically elements of the 
curriculum that use the city, retention 
of graduates in the area, research on 
urban topics and with urban partners, 
and economic impact. We are also 
looking at using a model developed by 
colleagues at IUPUI for assessing the 
scholarship of service (see Service 
1999). The model looks at four aspects 
of civic engagement: (1) its impact, (2) 
the intellectual nature of the work (e.g., 
command of expertise, innovation, in- 
terdisciplinary nature), (3) the sus- 
taining contribution (e.g., its devel- 
opmental complexity, whether leader- 
ship serves as a catalyst), and (4) com- 
munication through multiple and di- 
verse professional modes. 

The Uiban Universities Portfolio 
Project (UUPP) 

Can a portfolio be a performance indi- 
cator? What if, instead of creating a 
quantitative measure, an institution 
distributes a portfolio of student work, 
teaching, and other institutional activi- 
ties? Is that a performance indicator? 

IUPUI has developed an institu- 
tional portfolio through its participation 
in the Urban Universities Portfolio 
Project. 1 In it, we provide indicators for 
our three major mission components: 
Effective Student learning, Excellent 
Research and Scholarship, and Exem- 



plary Civic Engagement. Within stu- 
dent learning, the portfolio answers ba- 
sic questions such as who enrolls, what 
kind of programs support these stu- 
dents, and what evidence shows that 
the programs work. 

The section on our learning com- 
munities and their impact on students, 
for example, includes a blurb of a study 
examining the impact of participation 
on student progress and performance. 
The portfolio gives access to the whole 
report to anyone interested. 

An institutional portfolio can thus 
serve as a single campus indicator. 

As part of the UUPP project, we 
are meta-analyzing the portfolios of the 
six participating institutions to see 
what we have in common as urban 
universities. Our working themes in- 
clude access and support, diversity and 
pluralism, the urban context for stu- 
dent learning, the urban university 
as an intellectual resource, and civic 
engagement. 

In another project in which I’m in- 
volved — UUSPP, for Urban University 
Statistical Portrait Project (not to be 
confused with UUPP) — my colleagues 
and I are using the six themes that 
emerged from the UUPP meta-analysis 
as a starting point for developing meas- 
ures that represent the effectiveness of 
urban mission universities better than 
do current popular ones (e.g., retention 
and graduation rates). 

I hope you’ve noticed the recurrent 
theme in these institution-specific and 
consortia-based efforts I’ve just de- 
scribed. In all these examples, most of 
our time and effort is devoted to defin- 
ing and refining the broad themes (con- 
cepts) by which we think our perform- 
ance should be measured. In my 
opinion, far too much time is wasted 
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developing performance measures, 
without nearly enough thought given to 
what we are trying to measure. 

Emerging Approaches 

If you’ve been following the literature 
on performance indicators, you’ve 
probably read a lot lately about dash- 
board indicators and balanced score- 
cards. Personally, I don’t like the term 
“dashboard indicators,” because Fm not 
sure who’s driving, where we’re going, 
and whether we’re viewing this from 
the front seat or the back seat. It’s very 
much of a command-and-control orien- 
tation. It’s not yet clear whether dash- 
board indicators can be used for im- 
provement as well as for accountability. 

Balanced scorecards, as the name 
suggests, present a more balanced ap- 
proach. Rather than review balanced 
scorecards in detail here, I recommend 
that you read “Toward a Balanced 
Scorecard for Higher Education,” by 
Brent Ruben (1999) of Rutgers Univer- 
sity. Following Kaplan and Norton’s 
(1992) original formulation, Ruben pro- 
poses that a balanced scorecard incor- 
porate four perspectives: performance 
outcomes, constituency satisfaction, in- 
ternal business process, and learning 
and innovation. 

Ruben does a great job of spelling 
out the inadequacy of the “accounting- 
based” measures that have dominated 
performance indicators in higher 
education: 

• They are too historical; 

• They lack predictive power; 

• They can reward the wrong 
behavior; 

• They focus on inputs, not outputs; 

• They may not capture key busi- 
ness changes until it’s too late; 

• They relate to functions rather 



than cross-functional processes; 

• They give inadequate attention to 
difficult-to-quantify resources, 
such as intellectual capital. 

We’ve been aware of these concerns for 
years, and he expresses them very well. 

Ruben suggests that we should 
study the three basic areas that repre- 
sent the core of most of our missions: 
teachingflearning, scholarship/research, 
and service. He also suggests two other 
internal areas of focus to round out the 
balance: workplace satisfaction and fi- 
nances. As another example of his ad- 
aptation, Ruben describes two major 
themes for performance indicators of 
teachingdearning: the quality of prod- 
ucts (courses and programs) and stu- 
dent outcomes. 2 

A Few Concluding Precepts 

Returning to our original question: Is it 
possible to have a few good performance 
measures? A few good measures are, in 
fact, possible if you have a very focused 
mission and a very targeted clientele. 
The rest of us need more than a few 
good measures. 

Refining the criteria Trudy Banta 
and I proposed in our 1994 monograph, 
I offer the following characteristics for 
effective performance indicators: 

1. They should be designed using con- 
ceptually and methodologically 
sound procedures. 

2. They must be mission-related, re- 
flect core values, and aligned 
throughout your organization. In 
other words, they should reflect 
what your institution actually 
does in the way of planning and 
evaluation. 

3. They should be developed for tar- 
geted audiences. 
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Consider involving important con- 
stituents (faculty, students, employers, 
research partners, etc.) in the next it- 
eration of your mission statement and 
the indicators you will use to stimulate 
internal improvement and external 
accountability. 

I think you will find that you need 
to come up with not just a few good 
measures but a broad range of meas- 
ures that meet the needs of various 
constituent groups. You may end up 



with more than a few good measures, 
but you only need communicate them a 
few at a time. 

Finally, what’s the point of having 
performance indicators if they’re not 
used for continuous improvement. 
You’ve got to close the loop. In the final 
analysis, performance measures are 
only worthwhile if they can be used to 
help your unit or institution attain its 
goals and objectives. 



Notes 

1. AAHE and IUPUI are partners in this Pew-funded project, “The Urban Universities Portfo- 

lio Project: Assuring Quality for Multiple Publics.” The project engages six such universi- 
ties in the creation of institutional portfolios and in an innovative auditing process. The 
participating campuses are California State University-Sacramento, Georgia State Uni- 
versity, IUPUI, Portland State University, the University of Illinois-Chicago, and the 
University of Massachusetts Boston. For more, visit AAHE’s website at <www.aahe.org>. 

2. Since giving this presentation, I’ve combined some of Ruben’s ideas into IUPUI’s evolving 

performance indicator and institutional portfolio efforts. The latest model for IUPUI per- 
formance indicators is now portrayed in the “Performance Indicators” section of the 
IUPUI institutional portfolio at <dittp://www.imir.iupui.edu/iupuifolio>. 
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Assessment of 
Innovative Efforts: 
Lessons From the 
Learning Community 

Movement 

by Jean MacGregor, Vincent Tinto, 
and Jerri Holland Lindblad 



T he interest that’s brought the 
three of us together has been 
that strand of innovative work 
known as learning communities, par- 
ticularly the assessment of learning 
communities. Our remarks here will be 
framed, however, not only in the con- 
text of issues associated with learning 
community assessment but also as les- 
sons for the assessment of any innova- 
tive work. 

We will begin with a brief overview 
of curricular learning co mmuni ties to 
give you a sense of the water in which 
we all swim. We will then report briefly 
on research on the state of learning 
community assessment. We will follow 
that with some ways to frame assess- 
ment efforts that are particularly ap- 



propriate to innovative work. We will 
conclude with a few stories about 
learning community assessment and 
some lessons learned from them. 

What Is a Learning Community? 

The phrase “learning communities” is 
used in many contexts today, including 
individual classrooms as learning com- 
munities, learning communities in resi- 
dence life programs, and more recently, 
virtual learning communities. We will 
use the term here to represent curricu- 
lar approaches that link classes, often 
around interdisciplinary themes, and 
enroll common groups of students for a 
quarter, a semester, or in some cases 
even a year. We see such learning 
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communities as a very intentional re- 
structuring of time, ! space, and student 
life. 

There are multiple purposes for 
learning communities, and that’s why 
they’re a powerful kind of venue. The 
fundamental aim of learning commu- 
nities is to deepen learning, create a 
deeper sense of connection among ideas 
and curricular issues, and create a 
stronger sense of community. More spe- 
cifically, learning communities aim to 
promote the higher level of student en- 
gagement and success that comes from 
deeper intellectual interaction. Many of 
these programs aim to foster the com- 
plex thinking skills that come from ex- 
amining ideas in an interdisciplinary 
context. They also aim to foster a social 
as well as an intellectual community. 
Learning communities can be powerful 
for teachers as well: Teaching in a com- 
mon enterprise with common students 
can increase faculty vitality and en- 
hance repertoires of teaching practice. 

Learning communities put courses 
that are usually enrolled in and taught 
separately into larger, more coherent 
programs of study. This is not a new 
idea. We can find examples of these 
kinds of linkages going back to the 
1920s under a variety of names. In the 
last fifteen years, however, interest in 
expanding these kinds of programs has 
grown. In an interesting intersection, 
it’s in those same fifteen years that as- 
sessment has emerged on our campuses 
and become a priority. 

Because learning communities are 
developed at all kinds of institutions 
with all kinds of curricular configura- 
tions, it’s almost impossible to create or 
present a typology that includes all 
learning communities. But we can visu- 
alize a generalized set of types on a con- 



tinuum from simple to complex cur- 
ricular linkages, from minimal faculty 
coordination of the curriculum to a fully 
coordinated, team-taught program. 

In its simplest form, a learning 
community might be composed of a 
small group of students that travels to- 
gether, without faculty coordination, to 
stand-alone classes that also enroll 
other students. The student group often 
also meets in an additional, integrated 
seminar to discuss concepts addressed 
across the classes. For example, the stu- 
dents might take courses in history, 
film, and American literature, and the 
seminar might explore the question 
“What is the American character?” 

At the next step on the hierarchy, 
a learning community may be composed 
of a cohort of students that takes a pro- 
gram of two or perhaps three courses 
together. The courses are linked the- 
matically or by content, and the faculty 
teaching the courses plan the program 
collaboratively. Developmental students 
might, for example, take an inter- 
mediate algebra course linked to a pre- 
college chemistry course. At several in- 
stitutions, this “CheMath” program is 
preparatory to entering and passing col- 
lege chemistry. The algebra and chem- 
istry teachers work together to build 
basic concepts in chemistry and inte- 
grate that with algebraic thinking. 

An example from the top of the hi- 
erarchy would be a fully team-taught 
program of courses that is embedded in 
an integrated program of study. For ex- 
ample, “Ecological Systems of Puget 
Sound” would include coursework in 
terrestrial and marine ecology, history 
of the Northwest, and statistics and 
mathematical modeling. 

These examples show that learn- 
ing communities have enormous vari- 
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ability in structure and content and in 
their degree of curricular coordination 
and co-curricular connections. They also 
have enormous variability in their 
pedagogies. In any given learning com- 
munity, you might see intensive writing 
and peer editing, cooperative or collabo- 
rative learning, the use of technology, 
service-learning, and many kinds of in- 
tegrative assignments and projects. 

As with many other innovative ef- 
forts, learning communities involve a 
simultaneous change in both curricu- 
lum and pedagogy and often veiy dif- 
ferent expectations of student work. 
They’re complex innovations and com- 
plex interventions. Just tweaking a 
course a bit does not create a learning 
community; both the curriculum and 
pedagogy must be rethought. 

Lessons From Assessments of 
Learning Communities 

Like many innovations that are ambi- 
tious and labor- and resource-intensive, 
learning communities often find them- 
selves immediately in the spotlight of 
scrutiny and pressed to prove them- 
selves very quickly. Program innovators 
are therefore not only inventing new 
curricula and exploring new pedagogies 
but also often taking on evaluation re- 
sponsibility as well. 

Our colleague Faith Gabelnick of- 
ten refers to this as the “poet-critic ten- 
sion” in innovation. We ask faculty to 
invent, experiment, improve, and tweak 
in a new exploratory space. At the same 
time, we ask them to be critics, very 
quickly evaluating their work to prove 
its efficacy. Almost all learning commu- 
nity programs developed over the past 
fifteen years have been bootstrap, 
grassroots endeavors that have bubbled 
up among inventive, pioneering folks 



who are pressed simultaneously to jus- 
tify, evaluate, and prove. It’s been 
messy but exciting. 

In a review of the Washington 
Center’s compilation of seventy assess- 
ment studies of learning communities, 
we found some promising results: 

• Learning community students 
generally fare better academically, 
socially, and personally than those 
in comparison groups. This is espe- 
cially true for at-risk students, un- 
derrepresented students, and stu- 
dents who generally make Cs and 
Ds — veiy good news. 

• Learning community students’ 
learning goes deeper, is more inte- 
grated, and is more complex than 
that of students in comparison 
groups. 

• Learning community faculty make 
significant gains in personal, so- 
cial, and professional development. 

• The integration of academic and 
social life gives both faculty and 
students a sense of community. 

• Faculty and students both develop 
sensitivity to and respect for other 
points of view, other cultures, and 
other people. 

Our review also reveals gaps in 
what we know. First, we know little 
about learning community impact on a 
number of cohorts, including com- 
muters, transfers, alumni, and admin- 
istrators. We also need more infor- 
mation on underrepresented students, 
on faculty and student transitions in 
and out of learning communities, and 
on what brings about intended and 
unintended results. 

Second, more than half the studies 
examined the most-integrated type of 
learning community, generally called 
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“coordinated studies.” We need more 
research on the less-integrated types of 
learning communities, such as Fresh- 
man Interest Groups, linked courses, 
and course clusters. 

Third, we need more carefully 
planned studies that portray the spec- 
trum with what researchers call “disin- 
terested subjectivity.” 

Fourth, researchers and evaluators 
need opportunities to learn or re-leam 
qualitative and quantitative techniques. 
They also need time. Such opportunities 
are especially important in this era of 
performance-based funding. Sustained, 
rigorous, qualitative and quantitative 
assessment will contribute meaning- 
fully to the growth and viability of inno- 
vations such as the learning community 
movement. 

Planning an Assessment of an 
Innovative Program: Asking the 
Right Questions 

Let’s now use what we have learned 
about assessing learning communities 
to consider how to think about the as- 
sessment of such innovations. Unfortu- 
nately, when you’re planning a pro- 
gram, how you will assess it is often one 
of the very last questions you ask your- 
self. But it shouldn’t be your first ques- 
tion, either; a range of other questions 
often dictate what you end up doing and 
so should be considered first. 

The first question you must ask 
when planning an assessment of an in- 
novative effort is what mode you are in. 
Are you using assessment to improve 
the learning community? Or are you 
proving the value of the learning com- 
munity in order to validate and/or insti- 
tutionalize it? Those two modes give 
rise to very different ways of thinking 
about assessment. 



Second, what are the goals of the 
innovation? What are the specific aims 
that it intends to achieve? These will 
tell you what outcome measures to fo- 
cus on. 

Third, who are the audiences with 
whom you intend to share the data and 
with whom you must hold a con- 
versation about the evidence? If, for ex- 
ample, your mode is to prove and vali- 
date the learning community, you need 
to ask yourself whose support and par- 
ticipation you hope to gain. Faculty? 
Administrators? The president? State 
and foundation agencies? Other stu- 
dents or potential students? 

Fourth, what data do you want to 
collect, and what methods do you want 
to use to collect those data? If, for exam- 
ple, you intend to prove to potential 
students that the learning community 
is worth enrolling in, you would need 
qualitative information such as student 
quotes. If, however, you need to prove 
the learning community’s value to an 
a dminis trator who is concerned about 
student retention gains, you’ll need 
some numbers. 

Finally, what are the strategies you 
will use to share the data? You know 
the old saw: Data not used are data not 
worth collecting. You must plan now 
how the data will be shared, with which 
audiences, and in which mode, because 
that will determine how you assess the 
program. Only after you’ve answered 
these questions can you design the as- 
sessment of an innovative program. 

Now here are three examples to 
highlight the different ways learning 
community advocates have learned to 
do assessments for varying purposes. 

1. Seattle Central Community 
College. In 1985, Seattle Central 
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Community College began experi- 
menting ‘with team-taught, inter- 
disciplinary coordinated study pro- 
grams, and the immediate feedback 
was very positive. The faculty teaching 
in the programs were excited; a few key 
administrators were very supportive; 
students seemed stimulated and 
pleased. There was some impressive 
data about rates of course completion 
and re-enrollment. 

So all went well, in some ways. 
But the faculty teaching in the program 
wanted to convince themselves and 
their colleagues of the real texture of 
the program. Was it just a feel-good ex- 
perience, or was it something sub- 
stantive, of real value to students in 
their lives and their learning? 

The faculty were looking for some- 
thing more than the standard end-of- 
course, fill-in-the-bubble evaluations, 
which seemed unable to capture the 
complexity of what the faculty felt was 
happening in the classroom. They were 
also looking for a simple way to obtain 
information on students’ perceptions of 
their learning community experience 
without intruding on student or faculty 
time and without disrupting the learn- 
ing community’s curricular focus or 
ethos. 

The strategy the faculty struck 
upon was self-reflection. They asked 
students to write formal essays reflect- 
ing on their learning, not so much 
evaluating the program or the faculty 
but rather reflecting on themselves as 
learners, describing their experience 
and identifying benefits and challenges. 
In a blind study, volunteer faculty read- 
ers (not teaching in lea rning com- 
munities at the time) analyzed the es- 
says to discern patterns and themes. 

The results were quite positive. 



The readers found that students very 
much valued the learning community 
experience in just the ways that most 
learning community teachers hoped 
they would. There was a genuine reso- 
nance between teachers’ goals for the 
learning community and students’ iden- 
tification of those goals. 

The more important and lasting 
results, however, weren’t anticipated, a 
true lesson in evaluating innovative 
work. Students, on their own, without 
any prompting, regularly mentioned the 
very general-education outcomes the 
college had already identified as central 
to a degree. And the essays led to in- 
tense discussions among faculty on the 
language students used to talk about 
their experience, discussions that led to 
additional changes and improvements 
in the learning community program. 

So the self-reflection exercise gave 
students a valuable experience and 
gave faculty a unique window on that 
experience. Within a year, all learning 
community teams began to use student 
self-reflection essays systematically. 
These reflective writing assignments 
have become a permanent fixture in all 
learning community teaching at this 
college. 

A very modest effort to prove the 
efficacy of the program unwittingly be- 
came an enriched form of learning 
community pedagogy. A small, inex- 
pensive, grassroots assessment effort 
had a very high degree of payoff — 
many unexpected results. 

2. Spokane Falls Community 
College. Spokane Falls Community 
College had a well-established, well- 
regarded ten-year-old learning com- 
munity program with campus support. 
Then, a sudden, complete turnover of 
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senior administrators, a restructuring, 
a new focus on outcomes assessment, 
and an upcoming accreditation brought 
forth the occasion to do a major year- 
long study — a highly qualitative one — 
to address these constituencies and 
needs. 

A team of eight learning commu- 
nity faculty members from disciplines 
across the college was trained in quali- 
tative methodology. Using small, mobile 
research teams, they interviewed stu- 
dents, faculty members, and adminis- 
trators with a wide range of perceptions 
about learning communities. They ad- 
ministered questionnaires to learning 
community students and a control 
population in three types of classes over 
three consecutive quarters. They pored 
over student class journals. The institu- 
tional research office collected some 
quantitative data for them. 

The results affirmed the quality of 
the program and its significant contri- 
butions to the college: Faculty and stu- 
dents highly valued the richness of 
community they experienced. Students 
lauded the connections on and off cam- 
pus that their group work experiences 
encouraged. No matter what the level of 
student ability, both teachers and 
students alike highlighted the intellec- 
tual gains and depth of lea rning that 
all types of learning communities 
experienced. 

The team also identified weak- 
nesses and needs and made plans to 
address them: training in e ngaging 
students in seminar discussions, 
mentoring for newer faculty members, 
increasing the numbers of learning 
community offerings and assessments, 
providing more and better space for 
learning community programs, and 
deepening collegiality across the college 



by creating other community-building 
learning occasions. As an added benefit, 
the research team itself experienced a 
research process that mirrors the inclu- 
sive, collaborative, and conversational 
style of the learning community class- 
room. The study’s outcomes also re- 
educated eveiyone — students, faculty, 
and administrators — about an innova- 
tion that many had come to take for 
granted. 

3. Creating Powerful Assess- 
ments Through Multiple Methods. 

Multiple methods make for a more 
powerful assessment, one that not only 
helps document and explain the impact 
of a program but also allows us to dis- 
cover other goals, perhaps unintended, 
that might have more impact on the 
program than expected beforehand. So 
our third example is from a national 
study of several collaborative learning 
communities across the United States. 
We collected quantitative survey infor- 
mation from program students and a 
matched comparison sample, as well as 
qualitative evidence, including inter- 
views, observations, focus groups, and 
ethnographic research. 

We learned that quantitative evi- 
dence is often very important in dem- 
onstrating impact. Our survey instru- 
ment, a variant of Robert Pace’s Stu- 
dent Experience Questionnaire (1984), 
measured the amount of activity that 
students and faculty invest in the 
learning community, in the comparison 
groups, in course work, and in libraiy 
work. It also assessed faculty and stu- 
dent engagement in writing and their 
perceptions of gain. Across the board, 
students in learning communities in- 
vested significantly more “time on 
task.” The engagement clearly activated 
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their involvement. 

We also measured student percep- 
tions of the environment, classes, other 
students, faculty, administrators, cam- 
pus climate, and their own involve- 
ment. Again, across the board, students 
in learning communities perceived the 
environment more positively, them- 
selves as more involved, and even per- 
ceived administrators as nicer. (So if 
you’re an administrator, try learning 
communities!) 

We also examined pass rates in 
developmental courses and continua- 
tion or persistence. For each measure, 
there were significant differences favor- 
ing students in learning communities. 
This held true both for developmen- 
tal education students and for other 
students. 

While these quantitative data 
document the effects of learning com- 
munities, they don’t tell us why these 
effects occur. This is where qualitative 
studies help; they indicate the under- 
lying dynamics that help explain what’s 
happening. For instance, our qualita- 
tive findings showed that students find 
support through the activities that 
emerge from the collaborative exercises 
of the learning community. One student 
said, “The learning community was like 
a raft running the rapids of my life.” 
She had two children, two jobs, and no 
partner in the house. The support she 
found in a learning community was 
critical to her managing to continue. 

Some other quotations from our 
qualitative studies document how en- 
gagement leads to involvement in 
learning: “The more I talk to other peo- 
ple about the class stuff, the homework, 
the tests, the more I’m actually learn- 
ing.” “[I’m learning more] not only 
about other people but also about the 



subject, because my brain is getting 
more [and] because I’m getting more 
involved with other students in the 
class.” These quotations help explain 
why social engagement leads to learn- 
ing gains. 

One student adds, “I’m getting 
more involved with the class, even after 
class.” For many students, the class be- 
comes a peer group whose activities 
emerge from and expand beyond the 
classroom. George Kuh has a wonderful 
expression: a “seamless learning envi- 
ronment.” We found that learning 
communities and collaborative peda- 
gogies create seamless learning envi- 
ronments, expanding from the class- 
room out. 

In learning communities, students 
learn to interact with people of different 
races, sizes, colors, etc. As a result, not 
only do they learn more, they learn bet- 
ter. It isn’t just spending more time 
with others — it’s the value students 
place on this powerful, diverse environ- 
ment of learning from others. This is 
what students value most about a 
learning community. 

We didn’t expect to uncover the 
findings demonstrated by these quota- 
tions. As this example demonstrates, 
multiple methods, and qualitative 
methods in particular, allow you to not 
only explain but discover things you 
may not have thought of beforehand. 

Thoughts to Reflect On 

Two of these stories are examples of in- 
ternal assessment work that were 
grassroots in nature and had positive 
impacts within their colleges in terms of 
communicating results. They contrast 
with our third story, which stems from 
a major national research project using 
multiple methods to study a variety of 
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institutions. All three stories make 
clear that the struggles to make an in- 
novative program work and to prove 
that it works must happen side by side. 

There are several important les- 
sons from the information we’ve shared. 

• If any kind of innovative work is to 
expand and become more robust, 
we need to get much more serious 
about building both the practice 
and the assessment of that work. 

• If you’re not clear on the goals of 
an assessment and the audiences 
to which that assessment will be 
directed, it’s hard to do the assess- 
ment well. So your first task is to 
ask yourself why, with whom, and 
for what purpose you are assess- 
ing, and then proceed. 

Reference 



• Gathering data isn’t as important 
as using it strategically and com- 
municating it to both receptive and 
not-so-receptive audiences. 

• Finally, assessment work is never 
done. We can’t assume that once 
an innovation is in place, our work 
is complete and our innovation will 
continue to be fully understood. 
Senior administrators come and 
go, events occur, and change hap- 
pens, so the need to re-educate and 
re-develop will re-emerge! 

That means hard work, but that hard 
work brings forth the opportunity for us 
all to learn. 
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Accreditation — 
Where Credit Is Due 



by Barbara Wright 



T hink, for a moment, about what 
students do that you least like. 
Some of us hate when students 
ask, “Is this going to be on the test?” or 
“What do I have to do to get an A?” 
Some of us wish they would work 
harder and spend more time studying. 
Why do these things bother us so 
much? It’s because they embody a su- 
perficial, instrumental kind of approach 
to what we, as educators, consider the 
very serious business of learning. 

The sort of student we’re talking 
about here is not, alas, aiming for the 
joy of discovery, nor for the exquisite 
moment when he or she finally gets a 
diffi cult concept. Instead, the student 
seems to model rational choice theory 
with painful directness: How can I get 
the highest grade for the least invest- 
ment? As educators, we want higher 
learning to transform students’ hearts 
and minds. We want students to find 
the experience tr ans fixin g and trans- 
forming. Students, however, want to 
pass the course and get on with the rest 
of their fives. They know they can’t af- 
ford to rebel, so they give us compliance 
but not much more. 

What’s interesting is that we often 



don’t behave very differently from such 
students. When I do workshops on as- 
sessment, I often begin with a rather 
cynical but commonly held definition of 
assessment: Figure out what “they” 
want, find the quickest, least damaging 
way to respond, send off a report, and 
then forget it. 

The “they” requiring assessment is 
most often an accreditation agency. (A 
good 80 percent of the participants in a 
workshop I recently conducted for new- 
comers to assessment indicated that 
they were there at least in part because 
of accreditation.) Institutional response 
to accreditors is often, “What do you 
want from us? What do we have to do to 
get an A or at least not to fail?” 

The problem with this cynical 
definition, of course, is that it takes a 
superficial, compliance-oriented view of 
assessment as an activity that’s re- 
quired but has no meaning, no integ- 
rity, no connection to anything the 
institution values, such as deeper learn- 
ing or institutional transformation. 

I begin with this definition because 
I’m a great believer in the psychological 
value of getting problems out in the 
open. I also believe in the pedagogical 
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value of contrast. It is useful to talk 
about what the misconceptions are and 
how not to approach assessment before 
we get down to the hard work of think- 
ing about how to do it meaningfully and 
well. This enables our assessment to 
address our institution’s core questions 
and lead to improvement rather than a 
mere report. 

All of this reminds me of fractals, 
which I first encountered about five 
years ago in a book by Margaret J. 
Wheatley called Leadership and the 
New Science (Berrett-Koehler, 1992). 
Fractals are visual or physical phenom- 
ena that repeat a familiar pattern or 
design at ever smaller or larger lev- 
els of scale. They can be created 
through chemical reactions or through 
computer-generated repetition of math- 
ematical formulae. But they’re also eve- 
rywhere in nature: in cloud formations, 
in heads and branches and florets of 
broccoli, in trees and respiratory sys- 
tems, and rocky landscapes that repeat 
their geological formations infinitely in 
boulders, rocks, and particles of sand. 

Fractals are a useful metaphor for 
understanding the workings of organ- 
izational culture and for understanding 
why changing that culture can be so dif- 
ficult. Small, local deviations are up 
against the immense power of the over- 
riding design of the fractal-like culture. 
Conversely, if we can change the frac- 
tal’s — or the organization’s — design, 
there are repercussions throughout the 
structure. 

We often hear that the ultimate 
purpose of assessment, beyond improv- 
ing student learning, beyond under- 
standing how our programs work, is to 
change institutional culture. In accredi- 
tation circles and elsewhere we often 
hear about the need to develop a “cul- 



ture of inquiry” or a “culture of evi- 
dence.” More recently, we’ve heard 
about the need to become ‘learning or- 
ganizations.” (It’s been humbling to re- 
alize that, ironically, our institutions of 
higher learning haven’t been able to be- 
come learning organizations much more 
readily than banks or manufacturers or 
trucking companies have.) 

My aim here is to reflect on the 
ways in which assessment and accredi- 
tation have intersected over the past 
fifteen years, how they’ve helped each 
other to deliver, and the powerful effect 
that accreditation has had on the as- 
sessment movement. I will talk about 
what assessment has done for accredi- 
tation, what accreditation has done for 
assessment, and how I see the synergy 
working. Finally, I will address what I 
see as some of the limitations or dan- 
gers of this relationship, and close with 
a vision of where this dance may take 
us. 

The Intersection of Accreditation 
and Assessment 

The early- to mid-1980s were a time of 
widespread dissatisfaction with higher 
education, with flashy, sensational de- 
velopments like state mandates, public 
outcry about rising tuition costs, tales of 
things college graduates could not do, 
draconian state budget battles, and per- 
formance funding. The thinking of the 
times was encapsulated in reports 
originating from both inside and outside 
the academy: “A Nation at Risk,” “In- 
volvement In Learning,” “Integrity in 
the College Curriculum,” “Time for Re- 
sults,” and so on. The reports invariably- 
criticized the state of baccalaureate 
education and called for reform. 

Conspicuously absent from these 
reports was any discussion of the role 
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accrediting agencies had played or 
could play ‘in improving the quality of 
the undergraduate experience. Accredi- 
tors were stung. Apparently they had 
become the victim of their own invisi- 
bility and. discretion, thanks to their 
policies of confidentiality that prevented 
them from going public with either suc- 
cess stories or problem institutions. To 
make matters worse, accreditors fo- 
cused on surrogates of institutional 
effectiveness, namely resources and 
processes, under the assumption that 
these surrogates led to quality educa- 
tion. In the rough climate of the 1980s, 
this turned out to be a dangerous 
assumption. 

The contemporary postsecondary 
assessment movement began in earnest 
in 1985 with some early state mandates 
and AAHE’s first Assessment Con- 
ference. The number of attendees and 
sense of urgency in the air stunned con- 
ference organizers. Sensing that some- 
thing important was afoot, AAHE 
founded the AAHE Assessment Forum 
in 1987 with funding from the Fund for 
Improvement of Postsecondary Educa- 
tion (FIPSE). AAHE began to organize 
annual conferences, provide a much 
needed literature on assessment, and 
serve as a clearinghouse for ideas and 
information about model institutions. 

Also in 1987, the U.S. Department 
of Education established new criteria 
for the recognition of all accrediting 
bodies, calling for a focus on educational 
effectiveness. Regional as well as pro- 
fessional accreditors were now expected 
to consider information on educational 
effectiveness systematically, as part of 
the accreditation process, and to deter- 
mine whether institutions or programs 
“document the educational achievement 
of their students.” In this way, federal 



policymakers drew accreditation more 
directly into the process of holding in- 
stitutions accountable. 

The accreditation community’s re- 
sponse was far from perfunctory com- 
pliance. It seized the opportunity to 
redefine and revitalize itself and refocus 
its processes. In 1985 and 1986 — 
before the Department of Education’s 
new criteria were established — the 
Southern Association of Colleges and 
Schools (SACS) became the first re- 
gional association to adopt a major new 
standard on institutional effectiveness. 
Shortly thereafter, the Western Asso- 
ciation of Schools and Colleges (WASC) 
followed suit. In 1989, the North Cen- 
tral Association of Colleges and Schools 
(NCA) weighed in with a new policy re- 
quiring that all institutions assess stu- 
dent achievement as part of self-study. 

These associations were followed 
closely by the Middle States Commis- 
sion on Higher Education, the North- 
west Association of Schools and Col- 
leges, and finally the New England 
Association of Schools and Colleges 
(NEASC). 

All the regionals followed up with 
countless workshops, regional confer- 
ences, presentations at AAHE Assess- 
ment Conferences, and detailed manu- 
als designed to support the new push to 
assess. 

Recent Developments in 
Accreditation 

Even more has happened in a second 
wave of assessment-related accredi- 
tation activity from 1996 to the present. 
Following a flirtation with SPREs 
(“state postsecondary review entities”), 
the U.S. Department of Education 
ratcheted up the pressure in the 1998 
Higher Education Act reauthorization. 
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In the list of areas for which accreditors 
must have standards in order to meet 
federal compliance obligations, student 
academic achievement moved from 
ninth place to first. In other words, the 
federal government has deemed that 
academic achievement has primary im- 
portance, and this must be reflected in 
accreditation processes. 

The regionals have met this chal- 
lenge by taking yet more radical steps. 
Some very interesting experiments are 
in progress. WASC, for example, has 
proposed a new framework for accredi- 
tation that is organized around two core 
commitments, institutional capacity 
and educational effectiveness, examined 
in a twelve-year review cycle. 

The institutional capacity review, 
coming in the early years of the pro- 
posed cycle, will focus on some tradi- 
tional inputs, such as institutional poli- 
cies, structures, and resources, but will 
also ask how effectively these support 
teaching and learning. The goals are to 
develop more efficient models for pres- 
entation and evaluation of data, to keep 
data routinely updated and available 
(for example, on a website), and to. sig- 
nificantly reduce the labor and dollars 
that institutions now invest in collect- 
ing capacity data for the traditional 
self-study. 

The educational effectiveness re- 
view, a separate process, would occur at 
about the midpoint of the twelve-year 
cycle and focus on the institution’s edu- 
cational vision, its organization for 
learning, and evidence of effective stu- 
dent learning. The final six years of the 
cycle would be available for follow-ups 
as needed. 

WASC’s goals are to reduce the 
cost and cumbersomeness of the ac- 
creditation process but also to increase 



its usefiolness to the institution by 
building what WASC calls “institutional 
capacity” and by facilitating educational 
improvement. All participants in the 
process would be supported with special 
training and best practices for review- 
ing data and determining educational 
effectiveness. WASC is now in the proc- 
ess of rewriting its handbook of accredi- 
tation so that its standards are stream- 
lined and framed around these two core 
commitments. It’s a fascinating model. 

NCA has also undertaken initia- 
tives that push the envelope of tradi- 
tional accreditation practice. NCA re- 
cently conducted an exhaustive review 
of hundreds of self-studies in a sys- 
tematic effort to identify institutions 
with successful assessment programs 
and to understand what sets them 
apart from institutions with either less- 
successful programs or none at all. 

NCA has simultaneously launched 
its Academic Quality Improvement 
Project (AQIP), another reconceptuali- 
zation of the accreditation process. 
AQIPs goal is to make accredita- 
tion a more potent force in educa- 
tional reform by melding continuous 
quality improvement principles with 
accreditation. 

Institutions that sign on with 
AQIP will be expected to participate in 
required workshops, carry out assess- 
ments and other activities, and develop 
processes for continuous improvement. 
Because strengthening student learning 
is the primary purpose of an educa- 
tional institution, AQIP plans to de- 
velop a series of non-prescriptive ques- 
tions to help institutions to conduct 
investigations into student learning: 

“As long as an institution continues 

to participate in AQIP, working se- 
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riously to improve its processes and 
results, its accreditation will con- 
tinue. For participating insti- 
tutions, the formal reaffirmation of 
accreditation every seven years will 
be a simple validation process 
that should not require reports, 
team visits, or other costly or intru- 
sive processes” [for more, see 
www.aqip.org]. 

While participating institutions 
are expected to make a considerable in- 
vestment in AQIP activities, the payoffs 
are clear: The institution is clearly in- 
vesting in itself, not in an accreditation 
process to satisfy an external entity; 
and the cost and intrusiveness of tradi- 
tional accreditation are virtually elimi- 
nated. The AQIP and WASC initiatives 
share common goals. They both aim 
to make accreditation a more useful, 
less cumbersome, and less costly proc- 
ess for the institution, at a time when 
everyone, including accreditors, feels 
stretched to the limit by current work- 
loads. The initiatives also thoroughly 
embed assessment in these new 
frameworks. 

Other regional accreditors are en- 
gaged in equally serious, if less radical, 
endeavors. Middle States is rewriting 
its standards and at the same time re- 
vising its statement on characteristics 
of excellence. It is also engaged in an 
extensive and systematic effort to iden- 
tify and train educators in the region 
who have an interest in assessment and 
educational effectiveness but no par- 
ticular prior expertise in assessment or 
accreditation. These individuals are 
being trained both to serve as qualified 
visiting team members and also to 
bring their new expertise back to their 
home institution. 



SACS has a reputation, with its 
notorious “the institution must” state- 
ments, of being the most onerously pre- 
scriptive of all the regionals. A close 
reading of SACS’s guidelines reveals, 
however, that while institutions must 
do assessment, how they shall do it is 
not dictated. Examples or possibilities 
are merely suggested. Nevertheless, 
SACS is currently reviewing its stan- 
dards, with the expectation that they 
will become less prescriptive. As with 
WASC and North Central, another goal 
is to make accreditation more efficient, 
less intrusive, more clearly driven by 
improvement, and less compliance- 
minded. 

Taking a uniquely gentle incre- 
mentalist approach, NEASC surveyed 
its membership two years ago in order 
to better understand the state of assess- 
ment at New England colleges and 
identify their needs. The survey re- 
vealed considerable consternation and a 
strong desire to learn more. 

During the following year NEASC 
sponsored ten training workshops at- 
tended by more than a thousand par- 
ticipants, which in turn led to many re- 
quests for campus workshops. In 1999- 
2000, NEASC convened a task force 
to develop an academic assessment 
protocol. NEASC’s hope is that a 
straightforward, non-prescriptive set 
of improvement-oriented questions will 
help institutions implement appro- 
priate assessment efforts and collect 
documentation in an institutional as- 
sessment portfolio. 

Since 1992 the Northwest Asso- 
ciation of Schools and Colleges has not 
had a particularly high profile in as- 
sessment. But commissioners report- 
edly take their assessment policy very 
seriously, to the point that some 70 per- 
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cent of the region’s focused interim re- 
ports are required Solely because insti- 
tutions have failed to show they are 
meeting expectations for assessment. 
Northwest’s goals are to introduce 
more-flexible, less data driven, and 
more-qualitative approaches to assess- 
ment and to help institutions move 
from developing plans and collecting 
data to actually using data for change 
and improvement. 

There are also notable examples of 
professional accreditation associations 
rethinking their requirements in light 
of the assessment movement. The Ac- 
creditation Board for Engineering and 
Technology (ABET), the National Coun- 
cil for Accreditation of Teacher Educa- 
tion (NCATE), and AACSB-The Inter- 
national Association for Management 
Education all spring to mind. As with 
the regionals, these specialized accredi- 
tation associations have shifted their 
focus from resources and other inputs 
to outcomes. Not only has this helped 
improve programs; it also has ad- 
dressed criticisms from presidents that 
the focus by professional accreditors on 
resources was demanding too much of 
institutional budgets, to the detriment 
of liberal arts and non-accredited 
programs. 

The Impact of Assessment on 
Accreditation 

So what has assessment done for ac- 
creditation? The bottom line on the in- 
credible amount of activity I’ve de- 
scribed is that, with prodding from the 
U.S. Department of Education and the 
savvy cooperation of accreditors, as- 
sessment has proven to be an extraor- 
dinarily useful tool. It has allowed ac- 
creditation to zero in on the crux of the 
matter — student learning and edu- 



cational effectiveness. It has enabled 
accreditation to recast itself, give it new 
energy and purpose, make itself more 
effective, and increase its clout. 

Today assessment has provided 
the cross-fertilization, the energy, for a 
second generation of accreditation re- 
forms. Accreditation has never been 
more vital, more interesting, more 
service-oriented, or more useful to its 
member institutions than it is today. 
And it’s still on a roll. 

The Impact of Accreditation on 
Assessment 

Turning the tables, what has accredi- 
tation done for assessment? Clearly, 
just as assessment revitalized accredi- 
tation, accreditation’s insistence on as- 
sessment has kept the assessment 
movement alive and thriving. But ac- 
creditation has been a sleeper in this 
role, little noticed and something we 
hardly anticipated in the early years of 
the assessment movement. 

Some pundits felt that the assess- 
ment movement was at halftime in 
1990, with perhaps another five years 
left. Yet the flow of novices and the de- 
mand for training and information con- 
tinue unabated. The assessment 
movement clearly outlived the predic- 
tion of a ten-year life span and, like ac- 
creditation, is st ill going strong. 

To what can we attribute this lon- 
gevity? Clearly one factor is that assess- 
ment has been a point of confluence for 
a whole series of new ideas and issues 
in education: new understandings of 
teaching and learning, critiques of tra- 
ditional methods of testing and evalua- 
tion, new definitions of the disciplines, a 
search for durable knowledge in a time 
of epistemological instability, renewed 
concern about students’ affective and 
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values development, and on and on. It 
helps, too, that we have stunning exam- 
ples of success, such as Alvemo College 
and King’s College. 

I would argue, however, that the 
single most powerful contributor to as- 
sessment’s staying power has been its 
championing by regional and profes- 
sional accreditation. 

To finance activities related to as- 
sessment and educational effectiveness, 
accreditation agencies have received 
sizable grants from sources including 
FIPSE, Davis, Knight, and the Pew 
Charitable Trusts. This flow of millions 
of dollars in funding has supported as- 
sessment as well as accreditation. It 
has enabled accreditation to contribute 
incalculably to the creation of human 
capital in assessment, not only through 
assessment materials and activities but 
also by motivating individuals to attend 
events such as AAHE’s annual Assess- 
ment Conference. 

Accreditation has also provided the 
external push that can be exploited by 
institutional leadership to get its own 
internal assessment effort rolling and to 
motivate individuals to get that neces- 
sary training. It has provided the per- 
fect combination of requirement and 
freedom for the institution to define 
that requirement. In so doing, accredi- 
tation has reinforced assessment’s own 
principles of good practice, emphasizing 
that assessment has to be in tune with 
the mission and core values of the insti- 
tution, tailored to the local context, fo- 
cused on student learning, and used for 
improvement, not mere compliance. 

By changing its own emphasis, ac- 
creditation has shifted our conception of 
quality from one defined by reputation 
and resources to one defined by student 
learning outcomes and talent devel- 



opment, or value-added. In so doing, ac- 
creditation, like assessment, is making 
educational quality more inclusive, 
more democratic, more egalitarian. 

The Symbiosis Continues 

The pressure from accreditors to focus 
on assessment has been slow but steady 
over these last ten or fifteen years. The 
rhythm of five-year reports and decen- 
nial reaccreditation may seem so slug- 
gish as to be utterly ineffectual. But the 
rhythm has been inescapable, like the 
flow of a glacier. Watching a glacier 
move is about as exciting as watching 
grass grow, unless you’re in the right 
place at the right time and an iceberg 
breaks off and crashes into the ocean. 
But glaciers do move, and they are 
powerful if not flashy. In their path 
they transport boulders, scour valleys, 
carve new riverbeds. 

Likewise, while our attention is di- 
verted by the babbling brooks of day-to- 
day policy shifts, yellow journalism 
about what students can’t do now, or 
the latest rise in some college’s already 
remarkable room, board, tuition, and 
fees, accreditation just trudges along. 
While administrators, legislators, or 
state boards may become distracted by 
other issues, accreditation keeps com- 
ing back, even if it does take five or ten 
years. Some schools have been asked 
two or t hr ee times now what they are 
doing about assessment and why it’s 
taking them so long. They’re beginning 
to get the message that assessment 
matters and it isn’t going away. 

One of the sources of faculty resis- 
tance, not just to assessment but to 
many initiatives, is the certainty of 
many faculty that they will still be on 
campus long after the current admin- 
istrator and his or her initiative du jour 
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have moved on. But now the immov- 
ableness of faculty ’has met its match 
in the low-key but irresistible force of 
accreditation. 

You can see what has made the 
collaboration of assessment and accredi- 
tation so powerful. Accreditation and 
assessment haven’t merely worked to- 
gether on a discreet project. They have, 
instead, done something more complex 
and ambitious. They have begun to 
build a whole new infrastructure for 
teaching and learning, for improving it, 
and for being accountable for it. That 
takes time, decades. It progresses at a 
glacial pace, but eventually it trans- 
forms the whole landscape. 

Some Cautions 

Despite considerable success and a 
hopeful prognosis, accreditors do see 
distinct dangers on the horizon for their 
partnership with assessment. A major 
concern shared by virtually all the re- 
gionals is that assessment findings 
could be used inappropriately — tied, 
for example, to performance funding, or 
used punitively, all but wiping out the 
possibility for institutions to keep the 
focus on improvement. 

Policymakers and politicians are in 
search of simple answers to difficult 
educational questions and do not al- 
ways understand the uniqueness of in- 
dividual campuses, the complexities of 
teaching and lea rnin g, the difficulty of 
assessment, and the madness of taking 
a single test score as a complete and 
correct indicator of institutional quality. 
The task facing all of us is to keep get- 
ting the message out that such actions 
are irresponsible. 

Another concern is the possibility 
that assessment may become the new 
orthodoxy of accreditation, as formulaic 



as counting the number of books in the 
library and the number of PhDs on the 
faculty and as capricious and anecdotal 
as old-style fact-finding interviews. 
Both assessment and accreditation 
need to guard against that, too. 

New Challenges for Assessment 
and Accreditation 

Assessment has challenged accredita- 
tion to grow beyond its traditional role. 
It has given accreditation a way to zero 
in on the tough questions of student 
learning and educational effectiveness. 
Accreditation has evolved, and now it is 
challenging assessment to ratchet up 
its intensity and creativity. 

Accreditation has put serious 
money into assessment. What about 
campuses? Consciously or uncon- 
sciously, everyone follows the money, 
and campus folk have an exquisite sen- 
sitivity to what is really valued — and 
invested in and rewarded — as opposed 
to what receives eloquent lip service, 
period. 

If assessment is going to mean 
anything at all, it has to be recognized 
as a legitimate contribution to schol- 
arship and become a normal part of 
promotion and tenure considerations. 
Unfortunately, this is easier said than 
done. There appears to be an unwill- 
ingness to do this at the institutional 
level for both practical and more intan- 
gible reasons. 

First, a change in scholarship focus 
could mean the possible loss of research 
grant funding and the dollars from in- 
direct costs that grants bring in, dollars 
that research institutions in particular 
are very dependent upon. Second, 
there’s the fear that a change in schol- 
arship focus could lead to loss of intel- 
lectual prestige and hence attractive- 
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ness and with them competitive advan- 
tage, tuition, and development dollars. 
In other words, a decision to reward as- 
sessment may look to institutional 
leadership like unilateral disarmament. 

Similarly, faculty may be unwilling 
to take a chance on the scholarship of 
teaching, learning, and assessment, 
even if there is institutional support for 
it, fearing the resultant inability to get 
another position elsewhere, since tradi- 
tional research and publication is the 
coin of the realm. In other words, it’s 
unilateral disarmament on the personal 
level. And this begins to sound like an- 
other fractal. 

My point is that this is an oppor- 
tunity for a new collaboration between 
assessment and accreditation. Working 
together from macro and micro levels, 
they can push together toward a multi- 
lateral acknowledgment that assess- 
ment is indeed important scholarly 
work. 

In closing let me return to Mar- 
garet Wheatley and fractals. Like 
Wheatley, I believe fractals can teach 
us something interesting and impor- 
tant, not just about natural formations 
but also about human endeavors. For 
example, it’s impossible ever to know 
the precise measurement of a fractal. 
Wheatley explains the simple, seminal 
fractal exercise that Benoit Mandelbrot, 
to whom we owe the concept of fractals, 
presented to colleagues and students. 
He asked, “How long is the coast of 
Britain?” As his colleagues soon under- 
stood, there is no final answer to this 
question. The closer you zoom in on the 
coastline, the more there is to measure. 

Since there can be no definitive 
measurement, what is important in a 
fractal landscape is to note the quality 
of the system, its complexity and distin- 



guishing shapes, and how it differs from 
other fractals. If we ignore these quali- 
tative factors and focus on quantitative 
measures, we will forever be frustrated 
by the incomplete and never-ending in- 
formation we receive. What we can 
know and what is important to know is 
the shape of the whole, how it develops, 
changes, how it compares with other 
systems. 

These words contain an extraor- 
dinarily important lesson for us, for 
the assessment movement, for accredi- 
tation, for all the institutions and or- 
ganizations involved in postsecondary 
education. The cultural change that 
we seek cannot be achieved by mere 
measurement. 

But if we step back from data 
gathering to interpret those data quali- 
tatively, if we share our insights and 
work together to discern and define 
that new culture, if the communication 
and process of assessment that have 
proven so powerful on campuses can be 
carried out at higher levels of scale, 
then perhaps all of us together, from 
individuals or individual units or 
schools or colleges or campuses or asso- 
ciations or accreditors, will have a 
chance at transformation. We will have 
a new culture of assessment, one that 
focuses on the adventure of discovery, 
that is delighted by unexpected findings 
instead of terrified by them, that goes 
beyond data to develop narratives that 
capture the shape of what we’ve seen. 

Discovery, and the telling of that 
story of discovery, should be joyous, 
playful, surprising, maybe even funny. 
“Wouldn’t we all welcome more laugh- 
ter in the halls of management?” 
Wheatley asks toward the end of her 
book. Wouldn’t we all welcome more 
laughter in the halls of academe? 
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Imagine a world in which the fractal of 
drudgery and stress; of too much to do 
in too little time, is replaced at all levels 
— from students and faculty and li- 
brarians and administrators to the of- 
fices of accreditors — with the joy and 
laughter of discovery, of true, deep 
learning. 

Utopian? Perhaps. I’ve been ac- 



cused more than once of being a hope- 
less idealist. But at their best, it’s this 
utopia on which assessment and ac- 
creditation are triangulating. Who 
wouldn’t trade what is all too often 
bleakest higher education bureaucracy 
for a living, breathing, self-reflecting 
culture of teaching and learning? 
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Assessment at the 
Millennium: 
Now What? 

by Thomas A. Angelo, Peter T. Ewell, 

and Cecilia Lopez 

facilitated by Theodore J. Marchese 



A t least three sources motivate 
faculty and administrators to 
engage in the assessment of 
their students’ learning. The first 
source is accountability demands from 
both accreditation agencies and state 
government. The second is changing 
conditions around us, specifically grow- 
ing diversification in postsecondaiy 
education, both in delivery me chanisms 
and in the nature of our students. The 
third source is the shifting paradigm of 
teaching and learning, in which we are 
moving from a teaching-centered to a 
learning-centered approach and devel- 
oping new pedagogies, curricula, and 
technologies to meet student needs 
better. 

All three of these forces affect us 
all, no matter what our roles are, and 
they will impact assessment in many 



ways in coming years. We will discuss 
how these forces are changing and the 
implications of those changes for as- 
sessment, and we will conclude with 
some thoughts on the major lessons of 
the assessment movement to date that 
can guide us as we move into the new 
mille nni um. 



Accountability Demands 

Accreditation activities have historically 
focused on institutional self-studies and 
evaluation visits by teams of peers. 
What we need to work on now are 
improvement-oriented alternatives to 
this model that focus on what students 
know and can do. 

State governments are looking at 
performance measures and have al- 
ready implemented some. As states be- 
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come more sophisticated in identifying 
and using performance measures, their 
need will increase for comparable data 
sets across states. 

As our assessment and account- 
ability tools become more sophisticated, 
we may see more substantial conse- 
quences for failing to meet performance 
standards. The accountability paradigm 
will change to focus on institutional im- 
provement, particularly on student 
learning. 

Growing Diversification in 
Postsecondary Education 

Most notions of academic accountability 
and assessment are based on the in- 
creasingly outdated paradigm that most 
students are of traditional college-going 
age, make their studies their exclusive 
full-time focus, and attend just one es- 
tablished college or university for the 
duration of their academic career. 

This model fails to take into ac- 
count the increasing diversity of our 
student body. The “traditional” college 
student has virtually vanished. Today 
nearly all students are non-traditional 
in one way or another; most work and 
attend college simultaneously. Our stu- 
dents are prepared in many ways, at 
many levels, and our programs and as- 
sessment methodologies need to ad- 
dress that. 

This outdated model also fails to 
consider what we call the enrollment 
“swirl”: the increasing tendency of stu- 
dents to attend more than one institu- 
tion. Cliff Adelman’s (1999) study of at- 
tendance and degree attainment found 
that 58 percent of baccalaureate recipi- 
ents attended more than one institution 
and 22 percent attended more than two. 
This trend raises questions about qual- 
ity control and attribution of product. 



The implication for assessment is that 
we must develop instruments that can 
assess the contributions of each institu- 
tion toward the eventual degree. 

Finally, this old model fails to ac- 
knowledge the many new providers of 
postsecondary education, including for- 
profit institutions and corporate provid- 
ers, and that the line between tradi- 
tional and non-traditional providers is 
blurring. 

The Teaching/Leaming Paradigm 

The notion of a shift from a teaching- 
centered to a learning-centered ap- 
proach to higher education was intro- 
duced by Barr and Tagg in 1995. As 
with many educational movements, this 
shift is happening slowly, but the pace 
could pick up over the next few years. 

The new teaching/learning para- 
digm is “customer-centered” — i.e., stu- 
dents, employers, and other key con- 
stituents have a greater role in deciding 
what’s important to learn. Over the 
coming years, employers will continue 
to “up the ante” in their demands for 
employees with necessary skills. 

The new teaching/learning para- 
digm includes a move away from 
knowledge-based learning outcomes 
that focus on facts and principles to 
“deep learning” outcomes such as the 
ability to apply knowledge and critical 
thinking and other s kill s. Deep learning 
should be important to us all, because 
employers, legislators, other key con- 
stituents, and the students themselves 
all want us to develop students’ s kills 
and abilities; it’s what makes a college 
education worth the investment. But 
even more important is that today’s 
world presents incredible challenges 
that can’t be met by people who have 
superficial educations. The task before 
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us is to be able to assess the complexi- 
ties of deep learning. 

Implications of These Trends for 
Assessment 

All three of these major trends — ac- 
countability, diversification, and the 
teaching/learning paradigm — have 
implications for the theory and practice 
of assessment: 

Increased demands for account- 
ability mean that we need to better 
commu ni cate the results of assessment 
to our constituents, especially those 
right on our own campuses. 

Accrediting bodies need to find 
more effective ways to engage cam- 
puses. They need to shift the focus from 
assessment as an external mandate to 
assessment as a transformational tool 
and demonstrate the value of assess- 
ment to an institution and its faculty. 

States, meanwhile, need to focus on 
what really matters: student learning. 
They will become more interested in 
linking assessment results to resources 
and accountability. 

The increasing diversification of 
higher education implies that students, 
rather than institutions and programs, 
will become the primary objects of as- 
sessment, as they move through mul- 
tiple higher education venues. Tradi- 
tional academic credentials — grades 
and credit hours — are becoming less 
relevant and credible. 

Both employers and consumers are 
increasingly demanding “degrees with 
integrity” (i.e., with certifiable skills), 
and assessment will probably move in 
the direction of providing such certifica- 
tion. We need to create certifications of 
student core competencies, especially 
those developed over a series of courses, 
in ways that have meaning and credi- 



bility to employers. Authentic perform- 
ance assessments, credible across 
higher education sectors, will likely pre- 
dominate over standardized tests as we 
become more sophisticated at devel- 
oping them. 

The new teaching/learning para- 
digm also means that we must use as- 
sessment to help faculty and students 
optimize the teaching/learning process. 
We need more focus on the scholarship 
of teaching and learning. What kinds of 
technologies, for example, best promote 
what kinds of learning, under what cir- 
cumstances, and for what students? 
Under what circumstances do students 
need to work together in order to 
maximize learning? When do students 
need not come together at all? When 
is a lecture hall setting the most 
effective? 

We also need to use our evolving 
knowledge of assessment, teaching, and 
learning to raise our expectations of 
both faculty and students. Faculty need 
to learn how to do their own assess- 
ments rather than use externally im- 
posed tools. Students, meanwhile, won’t 
give us good assessment information 
until they not only understand how to 
complete assessments but also appreci- 
ate their value. 

Because “add-on” assessments can 
be so time-consuming, it is increasingly 
important that we embed assessments 
in our usual teachingdearning activi- 
ties. Assessments can be incorporated 
into capstone experiences, fin al exams, 
internships, and other learning activi- 
ties that demonstrate development of 
key skills and understanding. 

What Have We Learned? 

We would like to conclude with some 
thoughts on what we’ve learned in the 
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fifteen years that higher education has 
been focusing on assessment. 

1. Start with what you already 
have. Most institutions don’t realize 
the extent of assessment information 
already available. Begin your assess- 
ment effort, therefore, by rounding up 
assessment information you already 
have. Include “proto-assessments”: in- 
formation from internships, field expe- 
riences, and capstone classes in which 
students synthesize their learning and 
apply it to new situations. 

Look at information that’s avail- 
able but not well captured, such as fac- 
ulty comments on papers. What do 
those comments tell you about patterns 
of student strengths and weaknesses? 
Find points in the student’s academic 
program at which you already collect 
some information; could you use that 
opportunity to collect something more 
sophisticated? 

“Starting with what you already 
have” includes starting with what oth- 
ers have done. Take the time to find 
models in other programs and on other 
campuses, so you’re not continually re- 
inventing the wheel. 

2. Build on success. Assessment 
requires an atmosphere of trust and 
goodwill. In reality, however, assess- 
ments are often generated by issues or 
crises — hardly trust-building circum- 
stances. Use assessment to look first at 
successes and then build a shared lan- 
guage, trust, and goals. Find just one 
example of an assessment success on 
your campus. How and why does the 
program or pedagogy work so well? 
What lessons can be extracted and 
shared to build more successes? 



3. Focus assessment on what 
matters most. What is most important 
at your institution? This should be 
stated in your mission, and it probably 
includes improving student learning; 
but verify this by looking at where your 
institution spends its money. Usually 
what is most important to us is what 
we’re willing to spend time, energy, and 
resources on. 

4. Everyone needs to know 
what’s in it for them. Faculty and 
students need a common understan ding 
of what assessment is and why it’s im- 
portant. The support of the president, 
provost, deans, and board is critical; 
without their active and enthusiastic 
support, assessment will not happen. 

5. Treat assessment as a “core 
competence.” If your institution is 
going to be competitive, address the 
needs of diverse students, and use 
technology effectively, it must do as- 
sessment and do it really well. Faculty 
can be convinced of this if they under- 
stand that assessment is nothing more 
than a scholarly activity turned 
on themselves. What’s different is 
that it’s a collective, not individual, 
responsibility. 

6. Assessment requires a cul- 
ture shift. Assessment isn’t a once- 
and-done project and doesn’t provide 
definitive answers. It is, instead, a con- 
tinuous cycle of raising questions and 
finding some answers that raise more 
questions. 

Assessment also requires a differ- 
ent leadership style. If assessment is to 
be effective, the findings must be used 
to make decisions about programmatic 
directions, resource allocations, etc. We 
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can’t do things in the same old way. exciting ones that have great potential 

We’re in a new xnillennium with new to lead us to greater excellence, 

paradigms and new approaches — 
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Abstract of highlights of a plenary session at AAHE’s 1999 Assessment Conference . 



Assessing for Quality 

in Learning 



A psychology undergraduate once 
said, “Memorize the important 
points, and you’ll do all right on 
the test.” Another student once said, “I 
prefer multiple-choice questions; they’re 
just a matter of learning facts, and no 
analysis or critique required, which I 
find tedious. I also dislike writing and 
prefer to have a set of questions in front 
of me.” 

Both these students were short- 
changing themselves cognitively. They 
were using a “surface” approach to 
learning, getting by with minimal effort 
and using inappropriate learning meth- 
ods, such as memorization, to demon- 
strate mastery. 

As educators, we’re looking not for 
such superficial outcomes but for per- 
formance-type outcomes: learning that 
brings knowledge to life. We want to 
encourage our students to move from 
surface to “deep” approaches to learn- 
ing. Deep learning means using the 
high-level, abstract cognitive processes 
that we want our students to develop. It 
includes explaining, arguing, reflecting, 
applying knowledge to problems that 
aren’t in the textbook, relating new 
problems to established principles, and 



by John Biggs 

hypothesizing. Students who take a 
deep approach are fully engaged in 
learning. 

We want to see how far students 
can move into this level of under- 
standing, so our challenge is to design 
teachingdearning and assessment ac- 
tivities that engage our students in 
deep learning. 

The Chinese Paradox 

Our challenge as teachers is to provide 
teaching/leaming activities that sup- 
port deep learning, and to eliminate 
those activities that support only sur- 
face learning, such as an insistence on 
coverage. Consider a study of Hong 
Kong students in which I was involved. 
According to the theories my colleagues 
and I had developed, educators there 
seemed to be doing everything wrong. 
Classes were large, the atmosphere 
authoritarian, and the assessment sys- 
tem was run by an examination 
authority separate from the curriculum 
department. Students were taught not 
in their native tongue but in English. 

We would have predicted that 
these students would be high in surface 
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learning but low in deep learning. What 
we found was a paradox: Students from 
this culture were outperforming West- 
ern students. The reason was our West- 
ern insistence on coverage, which is the 
enemy of understanding. To go deep 
takes time. The Chinese mathematics 
curriculum is deep, while the American 
curriculum has been described as “a 
mile wide and an inch deep.” 

The Role of Learning Activities 

How can we help students engage in 
deep learning? I use a remark by Tom 
Shuell (1986) as a “meta-theory” — 
overall framework — for my work: 

If students are to learn desired out- 
comes in a reasonably effective 
manner, then the teacher’s funda- 
mental task is to get students to en- 
gage in learning activities that are 
likely to result in their achieving 
these outcomes. . . . What the stu- 
dent does is more important in de- 
termining what’s learned than 
what the teacher does. 

Determining our objectives and 
ensuring that they focus on deep 
learning s kill s isn’t difficult. What’s 
challenging is operationalizing those 
objectives to create teaching and 
learning activities that are true to the 
content and reasonably effective. Talk- 
ing to students is the most common 
method we use to engage students, but 
it probably isn’t the most effective. 

One possibility is to use problem- 
based learning techniques. Under this 
model, our objective is that students 
will be able to solve problems. Our 
teaching strategy is giving them prob- 
lems to be solved. They will need infor- 



mation in order to solve the problems, 
so they learn content as well as 
problem-solving strategies. How effec- 
tively they solve the problem is our 
assessment. 

Aligning Objectives, Teaching, and 
Assessment 

Providing appropriate learning activi- 
ties isn’t sufficient to engage students 
in deep learning. We know that stu- 
dents react differently to different as- 
sessment formats. When students study 
for multiple-choice tests, they most of- 
ten describe themselves as memorizing. 
When they complete other assignments, 
they more often describe themselves as 
reading in depth and applying what 
they’ve learned. When students of 
teacher education used portfolios to ap- 
ply psychology to teaching and reflect 
on their learning, they commented, “A 
portfolio is not merely an assignment to 
be handed in but a powerful learning 
tool for the student” and “My portfolio 
led me to think about many questions 
that I never think of.” 

Teachers have an ordered view of 
the educational process. First come our 
objectives: knowing what we’re going to 
teach. We then design our teaching to 
match our objectives, and finally we de- 
sign our assessments to match what 
we’ve taught. 

Students, on the other hand, view 
this backwards. They see the assess- 
ment first, then learn accordingly, then 
at last see the outcomes we are trying 
to impart. This “backwash” approach is 
fine if our assessments correspond to 
what we truly want our students to 
know. If we ensure that our objectives 
are buried in our assessment, our stu- 
dents will learn what we want them to 
learn. We need assessment tasks that 
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address the sorts of deep learning skills 
that we want to see. 

Quantitative and Qualitative 
Approaches to Assessment 

I’ve mentioned that learning objectives 
are central to successful assessment; if 
we get them right, the rest should plug 
in and follow. If this is so simple, why 
doesn’t it happen? 

A major reason is our quantitative 
approach to assessment. We reduce 
student performance to a measurement 
scale. We assume that knowledge 
comes in units, each, either correct or 
incorrect, all worth the same amount. 
We further assume that it doesn’t mat- 
ter which units you get right, as long as 
you get at least a certain percentage of 
them right. We also assume that the 
percentage index works across students 
and across subjects. These are all inap- 
propriate assumptions. 

Another assumption we make 
when we use measurement scales is 
that the characteristic being measured 
is stable. This leads us to focus more on 
test-retest reliability and items that 
discriminate than on whether the tests 
relate to the curriculum. 

To make matters worse, some in- 
stitutions and jurisdictions have expec- 
tations, if not requirements, that each 
faculty member will award a limited 
number of As and Bs. If a certain pro- 
fessor awards As to 60 percent of her 
class, instead of congratulating her we 
hear groans about slack standards. 

A final assumption about the 
measurement-oriented approach to as- 
sessment is that testing should be un- 
der standard conditions. Students learn 
under different conditions, however. 
Portfolios aren’t standardized but can 
do an excellent job showing what stu- 



dents have learned. 

For all these reasons, we need to 
move from a quantitative, measure- 
ment-oriented model to a qualitative, 
standards-oriented model. A standards- 
oriented model assumes that learning is 
cumulative but that it changes in struc- 
ture as it grows and as things relate to 
one another in more complex ways. Un- 
der this model, learning should be as- 
sessed in terms of where students are 
in the evolution of that structure, and 
the final grade should represent that. 

This model requires teachers to ar- 
ticulate the highest level of under- 
standing that can be reasonably expec- 
ted of students and how students 
should demonstrate that level of under- 
standing. Teachers can do this, but of- 
ten they aren’t used to thinking this 
way. One approach is to pull together 
data on the comments teachers make 
on student papers and agree on what 
each type of comment says about the 
level of understanding demonstrated in 
the student’s work. 

Levels of Understanding 

The key to creating assessments that 
promote deep learning is to have a clear 
framework for operationalizing desired 
levels of understanding. I use a hierar- 
chical model, the “SOLO Taxonomy,” 
that describes the growth of under- 
standing (Biggs & Collis 1982). 

At the pre-structural level, the 
student has seized on an irrelevant fea- 
ture of the task. This occurs early on in 
learning, where the student misses the 
point of the lesson. 

Then at the uni-structural level, 
the student picks up one relevant as- 
pect, such as identifying a correct fea- 
ture or carrying out a simple procedure. 

As the student continues to learn, 
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he or she gradually acquires more and 
more aspects, but ; does not connect 
them. The student can now enumerate, 
describe, list, combine, do algorithms, 
and perform serial skills. This describes 
the multi- structural level of learning; 
the student has collected a set of 
bricks, but there is no blueprint for the 
building. 

At the next level, the aspects are 
integrated, such that the student can 
show how everything comes together. 
The case is made and the phenomenon 
explained. The student can compare 
and contrast, analyze, explain causes, 
justify, relate, and apply. This is oper- 
ating at the relational level; the bricks 
become a building. The major outcomes 
of any program should be at least at the 
relational level. Not all our teaching 
need be relational, but we should teach 
our students strategies that help them 
develop relational skills, and our as- 
sessments should match the level of 
understanding we want. 

The final level of learning is the 
extended abstract level of under- 
standing. At this level, students can go 
beyond what’s been given them into ab- 
stractions in far domains. They can 
theorize, generalize, hypothesize, and 



reflect. With their mastery of their dis- 
cipline, they can question what is given 
and theorize about alternatives. This is 
what we want professionals in any dis- 
cipline to be able to do. We want doc- 
tors, for example, to be able to use what 
they’ve learned to treat diseases and 
disorders that they’ve never seen be- 
fore. By the end of an academic pro- 
gram, therefore, our grading should in- 
clude assessment tasks that involve 
some extended abstract thinking. 

The SOLO framework can be used 
to ensure that objectives and assess- 
ment tasks are aligned. 

Conclusion 

In order to assess for quality in learn- 
ing, we must make our objectives clear 
in terms of what students are to learn, 
and then build learning activities and 
assessment tasks that develop and 
promote those skills. 

Can we use a qualitative stan- 
dards-based system? I believe so, and 
have described in detail how the align- 
ment model can be used for designing 
university and college teaching in my 
latest book (Biggs 1999). 
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Testing 

Disadvantaged Students: 
The Elusive Search for 



F airness is an important but elu- 
sive value in testing. I will 
describe several definitions of 
fairness in the context of testing and 
discuss their usefulness to us as we 
strive to improve testing products and 
how those products are used. I will con- 
clude by suggesting a reasonable defi- 
nition of fairness that we can all use. 

It’s important to begin by noting 
that all information we use to make de- 
cisions about students should reach a 
standard of fairness, including grades, 
letters of recommendation, and the es- 
says submitted with college applica- 
tions that may have been prepared with 
the guidance of a professional coun- 
selor. None of these, by itself, offers a 
completely fair consideration of what 
every student knows. 

What Is a Good Test? 

Good tests involve fairness from their 



What Is Fair 

by Sharon Robinson 

very beginnings, in their design, con- 
struction, and administration. Good 
tests reflect the diversity of voices of all 
test takers. They undergo a rigorous re- 
view in which the questions are judged 
to be fair. Good tests have standard 
administration procedures, so the 
scores reflect students’ unassisted per- 
formance. Scoring is as objective as pos- 
sible. If scoring must be subjective, it is 
not influenced by student background. 
Finally, good tests have evidence of re- 
liability and validity. We should require 
nothing less. 

Defining Fairness as a Lack of 
Group Differences 

In the first definition of fairness that I 
present to you, differences across 
groups are taken to be proof of test bias. 
It’s not possible to defend this definition 
rationally; if this definition is correct, 
tape measures, scales, and other objec- 
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tive measures are “biased” because they 
show differences among groups. Group 
differences alone are not proof of test 
bias, because a test may reveal persis- 
tent, real, and important differences. 

We know, for example, that the 
most effective way that high school stu- 
dents can improve their SAT scores is 
to take rigorous academic courses dur- 
ing their high school careers. The effect 
of such a curriculum outperforms all 
other strategies to improve test scores. 
But what about students who don’t 
have access to a sufficient number of 
rigorous high school courses? We want 
to eliminate group differences, but the 
only way to do that is by improving ac- 
cess to the knowledge and s kill s that 
the test measures. 

Defining Fairness by How the 
Questions Look 

Another way to define fairness is by 
looking at how the questions manifest 
themselves. Some questions may in- 
clude material that is not required by 
the purpose of the test but that some 
people find upsetting or provocative. 
For example: 

• A question on an American history 
test that mentions “pioneers and 
their wives” would be offensive, be- 
cause it implies that the wives 
were not pioneers. 

• A reading passage that mentions 
“people from Spain who estab- 
lished the first settlement in Mex- 
ico” would be offensive, because it 
disregards the indigenous people 
already there. 

• A question on the SAT on abortion 
would be unduly provocative, al- 
though on a medical exam it might 
be necessary. 

• A question about slavery or con- 



centration camps might be pro- 
vocative, unless the test is about 

history. 

Questions that exclude groups who 
would not have experience with the ma- 
terial (for example, questions on tax- 
free bonds or the game of polo) are in- 
appropriate, unless the material is es- 
sential to what is being tested. Also in- 
appropriate are questions that are 
intimidating or off-putting to some 
groups or that require regional knowl- 
edge. Students in Vermont may think of 
a muffler, for example, as something to 
wear, but students in New Mexico may 
think of it only as a component of a car. 
Tests are not bags of tricks; they are, 
rather, opportunities for students to 
show what they know. 

In reality, there is little connection 
between judgements about how fair test 
questions appear to be and how the test 
actually performs with various groups. 
It is nonetheless important to eliminate 
even the appearance of unfairness. 

Defining Fairness by Differences in 
Selection 

This third definition is an opportunity 
to illustrate the incredible complexities 
we deal with when we discuss fairness. 
It also shows how definitions of fairness 
can contradict one other. 

Many tests are used to predict fu- 
ture performance. The SAT, for exam- 
ple, is used to predict freshman grade 
point average (GPA). Higher scores 
tend to predict higher grades, and lower 
scores tend to predict lower grades. We 
use a statistical tool called regression 
analysis to make this prediction. If a 
particular group achieves lower grades 
than the regression analysis predicts, 
the test over-predicts those students’ 



ERIC 



70 — .Assessment To Promote Deep Learning 



75 



performance and unfairly advantages 
that group. Similarly, if a particular 
group achieves higher grades than the 
regression analysis predicts, the test 
under-predicts those students’ per- 
formance and unfairly disadvantages 
that group. 

I will use two common models of 
fairness to illustrate the difficulties in 
determining whether or not a test is 
fair. Under one model of fairness, a test 
is fair if it neither under-predicts nor 
over-predicts the performance of any 
relevant group. It is fair to individuals 
because their membership in a group is 
not a consideration in predicting their 
future performance. This definition 
doesn’t help us, however, to increase 
the representation of underrepresented 
groups. 

Another model of fairness suggests 
that we should add points to the scores 
of students in disadvantaged groups. 
Such models of fairness purposefully 
over-predict the scores of students in 
underrepresented groups to achieve 
some desired level of group equity. 

In a psychometric sense, whether 
or not a test vised for selection is fair 
depends on the model of fairness that is 
used. A test that is fair according to one 
of the models of fairness will be unfair 
according to the other model. 

Defining Fairness Through Validity 

Validity is the extent to which a test 
meets its intended purpose and infer- 
ences and decisions based on it are ap- 
propriate and backed by evidence. Un- 
der this definition, a question is fair if 
group differences are caused by valid 
aspects of the question. A test question 
is unfair, however, if it measures an as- 
pect of the candidate unrelated to the 
purpose of the test. 



For example, a mathematics ques- 
tion that contains difficult reading ma- 
terial may be unfair to people who have 
difficulty reading, because the purpose 
of the question is to measure math 
knowledge rather than reading ability- 
Reading material of the same level of 
difficulty may be perfectly fair if the 
purpose of the question is to measure 
reading ability. Because validity de- 
pends on purpose, in order to judge 
fairness under this definition, it is es- 
sential that we know what we want to 
find out from the test and then judge its 
validity in relation to that purpose. 

How Should We Judge Whether a 
Test Is Fair? 

You can see that the same test may be 
judged fair or unfair depending on 
which definition we apply. What steps 
can we take to ensure that the tests we 
are using are truly fair? 

• Become an aggressive and in- 
formed consumer. Test publishers 
should be prepared to talk about 
the steps they have taken to en- 
sure fairness, and they should pro- 
vide documentation of their efforts. 

• If the test itself is disclosed, look at 
its questions yourself. Do they 
seem fair? Ask the test publishers 
how they checked for fairness. 

How does each question perform 
among and between different 
groups? 

• Understand the purpose of the 
test. If it is used for professional li- 
censing or certification, is there a 
connection between the content of 
the test and what the job requires? 
If it is used for graduation, is there 
a connection between the content 
of the test and what was being 
taught? If there is a “cut” or pass- 
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ing score, how was it set? Some 
users set sliding “cut” scores based 
on supply and demand; that’s in- 
valid. 

• Does the administration procedure 
make the test accessible under 
standard conditions to all? 

There are no easy answers about 
validity and no perfectly valid test. Peo- 
ple will disagree about how much va- 
lidity evidence suffices. 

So what is the best definition of a 
“fair” test? While it’s important to check 
all sources of information for fairness, 
the best definition of fairness is validity. 
Ask yoturself, “Will information from 
this test allow me to make a better- 
informed decision about this person? 
Are differences among scores based on 
important differences that inform my 
decision?” 



Those who find access to a quality 
education elusive for some groups can 
help them find a chance of achieving it 
— and thereby performing optimally on 
tests — by following the three Ts: 

1. Tell what is going to be taught. In- 
volve diverse voices that contribute 
to defining the knowledge we value. 

2. Teach what we will be testing. Give 
every student an opportunity to 
learn what we value. 

3. Test the material that we teach; 
make sure that is what we test. 

When we take these three steps, we will 
finally break the cycle of elusiveness of 
fairness in the way we test. 

I welcome you to this call. 



Sharon Robinson is senior vice president and chief operating officer of Educational 
Testing Service. 
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More information and insight from past AAHE Assessment Conferences... 

Architecture for Change: Information as Foundation 

Major speeches from the 1998 conference on assessment’s contributions to improvement 
and accountability; the relationship of faculty to accreditation; invigorating learning 
through appropriate assessment; assessing programs, institutions, and powerful pedago- 
gies; and choosing questions, methods and ways to communicate results most effectively. 

Speakers that year were: Bruce M. Alberts (National Academy of Sciences); Judith 
Eaton (Council for Higher Education Accreditation); Stephen C. Ehrmann (The 
Flashlight Project); Jean MacGregor (National Learning Communities Dissemination 
Project); Margaret A. Miller (AAHE); Sue Rohan (Baldrige National Quality Award 
Office); and Jon. Wergin (Virginia Commonwealth University). 

Assessing Impact: Evidence and Action 

Major speeches from the 1997 conference from Ellen Earle Chaffee (Mayville State 
Univei-sity and Valley City State University) on “Listening to the People You Serve”; Peter 
T. Ewell (National Center for Higher Education Management Systems) on “Accountability 
and Assessment in a Second Decade: New Looks or Same Old Story?”; Sherril B. Gelinon 
(Portland State University) on “Intentional Improvement: The Deliberate Linkage of 
Assessment and Accreditation”; George Kuli (Indiana University Bloomington) on 
“Working Together to Enhance Student Learning Inside and Outside the Classroom”; Ted 
Marchese (AAHE) on “The New Conversations About Learning”; Margaret A. Miller 
(AAHE) on “Looking for Results: The Second Decade”; and Grant Wiggins (Center on 
Learning, Assessment, and School Structure) on “Feedback: How Learning Occurs.” 

To order these or other AAHE publications, request a catalog or visit AAHE’s website. 
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